Intelligent proposition method, system, device and storage medium simulating expert proposition

Abstract

The application relates to the technical field of artificial intelligence, and particularly provides an intelligent proposition method, system and device simulating expert proposition and a storage medium, which comprises the following steps: constructing a structured knowledge network integrating knowledge points, cognitive levels and proposition specifications; establishing a material vector database associated with a knowledge point system; extracting proposition constraint information from the knowledge network according to a target knowledge point, searching for related materials from the vector database, combining the materials into a generative prompt input large language model, and generating a test question draft; inputting the test question draft into at least one large language model serving as a simulation answering agent, evaluating the test question quality by analyzing the answering process and result, and outputting a qualified test question. The application can significantly improve the proposition efficiency and consistency, reduce the absolute dependence on expert experience, and realize the automatic output of high-quality test questions.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an intelligent question-setting method, system, device, and storage medium that simulates expert question setting. Background Technology

[0002] Traditional test question creation relies heavily on expert experience, resulting in low efficiency, high costs, and difficulty in ensuring consistent quality. While automated test question creation technology based on large language models has emerged, it typically employs an end-to-end generation model, lacking a deep integration of educational measurement rules and subject knowledge systems. This can easily lead to content "illusions," logical contradictions, questions exceeding syllabus limits, or uncontrolled difficulty. Furthermore, existing methods primarily focus on generation, lacking automated, simulation-based evaluation mechanisms for test question quality, making it difficult to ensure the rigor and effectiveness of the questions. Therefore, there is an urgent need for an automated test question creation method that can simulate expert question-creating logic, deeply integrate structured knowledge, and intelligently verify the quality of the generated results. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides an intelligent question-setting method, system, device and storage medium for simulating expert question setting, so as to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides an intelligent question-setting method that simulates expert question setting, comprising:

[0005] Based on multiple sets of data, a structured knowledge network containing a knowledge point system and its attribute relationships is constructed.

[0006] Collect the materials for the questions and represent them in vector form, and establish a material vector database associated with the knowledge point system;

[0007] Based on the target knowledge points, proposition constraint information is obtained from the structured knowledge network, and relevant materials are retrieved from the material vector database; the proposition constraint information and the relevant materials are combined into generative prompts, which are then input into the large language model to generate the initial draft of the test questions;

[0008] The initial draft of the test questions is input into at least one large language model that acts as a simulated answer agent to obtain its solution process and results; based on the solution process and results, the quality of the initial draft of the test questions is evaluated, and qualified test questions are output.

[0009] In one optional implementation, a structured knowledge network containing a knowledge point system and its attribute associations is constructed based on multiple propositional basis data, including:

[0010] Obtain various data sources for setting exam questions, including at least the exam syllabus, electronic textbooks, and past exam papers.

[0011] The examination syllabus is analyzed to extract knowledge points, cognitive levels, and ability requirements, and the weights of knowledge points are dynamically calculated based on semantic association.

[0012] The electronic teaching materials are analyzed to extract the core content paragraphs corresponding to the knowledge points;

[0013] The past exam questions were analyzed to construct structured data representing the relationship between questions, knowledge points, cognitive levels, and difficulty.

[0014] Based on the extracted knowledge points, weights, core content paragraphs, and structured data, a knowledge graph containing nodes, attributes, and relationships is constructed and stored. Graph reasoning models are used to infer and complete the logical dependencies between knowledge points.

[0015] In an optional implementation, based on the extracted knowledge points, weights, core content paragraphs, and structured data, a knowledge graph containing nodes, attributes, and relationships is constructed and stored, including:

[0016] Define multiple types of nodes, including at least: KnowledgePoint nodes representing knowledge points, TextbookChapter nodes representing textbook chapter content, PastQuestion nodes representing historical test questions, PropositionRule nodes representing question-setting rules, and CognitiveLevel nodes representing cognitive ability levels.

[0017] Define the relationship types connecting the nodes, including at least: the BelongsTo relationship connecting knowledge points and textbook chapters, the Examines relationship connecting historical test questions and knowledge points, the Requires relationship connecting knowledge points and cognitive ability levels, the Follows relationship connecting knowledge points and question specifications, and the Prerequisite relationship used to represent the logical dependencies between knowledge points.

[0018] Prerequisite relationships between knowledge points are inferred and completed using graph reasoning models;

[0019] The KnowledgePoint node has a dynamic weight attribute, and its weight value is calculated based on multi-factor weighting. The factors include at least: frequency factors obtained based on the statistical analysis of the historical test data, association factors obtained based on semantic similarity calculation, and hierarchical factors determined based on the cognitive ability level.

[0020] In an optional implementation, a graph reasoning model is used to infer and complete the prerequisite relationships between knowledge points, including:

[0021] Based on the existing nodes and relationships in the knowledge graph, at least one pair of observation features between knowledge points are extracted, and a probabilistic reasoning model is used to calculate the first probability value of the existence of a prior knowledge relationship between the knowledge point pairs.

[0022] The knowledge graph is subjected to graph representation learning. The neighbor information of the nodes is aggregated through message passing to update their vector representation. Based on the updated node vector representation, a second probability value is calculated through a link prediction model to determine whether there is a prior knowledge relationship between the knowledge point pairs.

[0023] By combining the first probability value and the second probability value, a comprehensive confidence level is obtained to indicate that there is a prior knowledge relationship between the knowledge point pairs. When the comprehensive confidence level exceeds a preset threshold and the relationship does not exist in the graph, a corresponding prior knowledge relationship is automatically created.

[0024] In one optional implementation, the test material is collected and vectorized to establish a material vector database associated with the knowledge point system, including:

[0025] Original proposition material texts are collected from at least one source of material, and the original proposition material texts are subjected to topic identification and summary generation to obtain structured material fragments;

[0026] The structured material fragments are encoded into fixed-dimensional vector representations using a pre-trained text embedding model.

[0027] Based on at least one pre-association strategy, the preliminary association between the structured material fragment and one or more knowledge points in the knowledge point system is determined, and the knowledge point identifier representing the preliminary association is stored as metadata corresponding to the vector representation.

[0028] In an optional implementation, based on the target knowledge points, propositional constraint information is obtained from the structured knowledge network, and relevant materials are retrieved from the material vector database; the propositional constraint information and the relevant materials are combined into generative prompts, which are then input into a large language model to generate a draft test question, including:

[0029] Based on the target knowledge points, propositional constraint information is extracted from the structured knowledge network. The propositional constraint information includes at least the cognitive level, difficulty requirements, and applicable question type rules of the target knowledge points.

[0030] Based on the target knowledge points, at least one semantically related material fragment is retrieved from the material vector database;

[0031] The proposition constraint information, the at least one material fragment, and the preset proposition task instructions are formatted and combined to construct generative prompt text;

[0032] The generative prompt text is input into a large language model. By adjusting the generation parameters of the large language model to control the output characteristics, initial drafts of test questions that conform to the propositional constraints are generated in batches.

[0033] In an optional implementation, based on the solution process and results, the quality of the initial draft of the test questions is evaluated, and qualified test questions are output, including:

[0034] Based on multiple preset quality dimensions, the solution process and results generated by the simulated answer agent are analyzed to generate scores for the initial draft of the test questions on each quality dimension.

[0035] Based on the scores across each quality dimension, the overall quality score of the initial draft of the test questions is calculated.

[0036] The overall quality score is compared with a preset acceptance threshold to determine whether the initial draft of the test questions meets the standards.

[0037] Secondly, the present invention provides an intelligent question-setting system that simulates expert question setting, comprising:

[0038] The knowledge graph construction module is used to construct a structured knowledge network containing knowledge point systems and their attribute relationships based on multiple propositional data.

[0039] The material processing module is used to collect the proposition materials and represent them in vector form, and to establish a material vector database associated with the knowledge point system.

[0040] The question generation module is used to obtain proposition constraint information from the structured knowledge network based on the target knowledge points, and retrieve relevant materials from the material vector database; combine the proposition constraint information and the relevant materials into generative prompts, and input them into the large language model to generate the initial draft of the question;

[0041] The test question evaluation module is used to input the initial test question into at least one large language model that acts as a simulated answer agent, obtain its answer process and results for the test question; based on the answer process and results, evaluate the quality of the initial test question and output qualified test questions.

[0042] Thirdly, a device is provided, comprising:

[0043] The memory is used to store the intelligent question-setting program that simulates expert questions;

[0044] The processor, when executing the intelligent proposition program that simulates expert propositions, implements the steps of the intelligent proposition method that simulates expert propositions as provided in the first aspect.

[0045] Fourthly, a computer-readable storage medium is provided, on which an intelligent problem-solving program simulating expert problem-solving is stored, wherein when the intelligent problem-solving program simulating expert problem-solving is executed by a processor, it implements the steps of the intelligent problem-solving method simulating expert problem-solving provided in the first aspect.

[0046] The beneficial effects of this invention lie in the fact that the intelligent question-setting method, system, device, and storage medium for simulating expert question setting provided by this invention, through the construction of a structured knowledge network and a vectorized material library, achieve deep semantic organization and efficient retrieval of the question-setting basis and materials, fundamentally standardizing the knowledge scope and logical foundation of the questions. By combining question constraints and related materials into generative prompts, it guides a large language model to generate accurate and controllable test questions, effectively suppressing "illusions" and improving content relevance. The innovative introduction of a simulation-based answer evaluation mechanism based on a large language model enables automated, multi-dimensional quality verification of question logic, difficulty, and effectiveness, forming an intelligent closed-loop control of question-setting quality. Ultimately, while significantly improving question-setting efficiency and consistency, it reduces the absolute dependence on expert experience, achieving the automated production of high-quality, large-scale test questions. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0049] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0053] The intelligent question-generating method for simulating expert question generation provided in this embodiment of the invention is executed by a computer device, and correspondingly, the intelligent question-generating system for simulating expert question generation runs on the computer device.

[0054] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be an intelligent question-setting system that simulates expert question setting. Depending on different needs, the order of steps in this flowchart can be changed, and some can be omitted.

[0055] like Figure 1 As shown, the method includes:

[0056] S1. Based on multiple sets of data on propositions, construct a structured knowledge network that includes a knowledge point system and its attribute relationships;

[0057] S2. Collect the test material and represent it in vector form, and establish a material vector database associated with the knowledge point system;

[0058] S3. Based on the target knowledge points, obtain proposition constraint information from the structured knowledge network and retrieve relevant materials from the material vector database; combine the proposition constraint information and the relevant materials into generative prompts, and input them into the large language model to generate a draft of the test questions;

[0059] S4. Input the initial draft of the test questions into at least one large language model that acts as a simulated answer agent, and obtain its answering process and results for the test questions; based on the answering process and results, evaluate the quality of the initial draft of the test questions, and output qualified test questions.

[0060] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0061] S101. Obtain data supporting various propositions.

[0062] The system collects structured and unstructured raw data from designated educational resource platforms or local storage via file upload interfaces or web crawlers. The data used to formulate the exam questions includes at least: exam syllabi stored in text or tabular form, electronic textbooks stored in PDF or Word format, and a database of past exam questions containing questions and answers. This data constitutes the basic raw materials for constructing the knowledge system.

[0063] S102. Analyze the examination syllabus and calculate its weights.

[0064] The system uses a pre-trained language model based on the Transformer architecture (such as BERT) to perform deep semantic parsing of the exam syllabus text. First, through named entity recognition and dependency parsing, it extracts discrete knowledge point entities and their corresponding cognitive levels (such as memory, understanding, application, and analysis) and ability requirement descriptions. Then, the system calculates the semantic correlation between knowledge points (e.g., using word vector cosine similarity) and dynamically assigns an initial weight to each knowledge point based on expert prior rules. This weight can be updated in subsequent steps based on the analysis results of past exam questions.

[0065] S103. Analyze and extract content from electronic teaching materials.

[0066] For electronic textbook documents, the system first performs format parsing and text cleaning to obtain plain text content. Then, it applies graph model algorithms such as TextRank or unsupervised summarization models to analyze the text within each chapter or knowledge unit, identifying and extracting core content paragraphs expressing core concepts, while discarding redundant, illustrative, or transitional text. The extracted core paragraphs are mapped and associated with the knowledge points extracted in S102 through chapter titles, keywords, etc.

[0067] S104. Analyze and structure past exam questions.

[0068] The system analyzes past exam questions question by question. Natural language processing technology is used to identify the stem, options, answer, and explanation text for each question. Through matching or classification models, each question is associated with the knowledge points defined in S102, and its cognitive level and difficulty value are labeled and statistically derived from historical score data. Finally, a structured data entry is constructed for each question, clearly representing the four-element relationship between "question-knowledge point-cognitive level-difficulty," forming a database of question-setting patterns.

[0069] S105. Based on the extracted knowledge points, weights, core content paragraphs, and structured data, construct and store a knowledge graph containing nodes, attributes, and relationships, including:

[0070] Knowledge graph structure definition and storage implementation:

[0071] The system uses an attribute graph model to formally represent structured knowledge networks. In the graph database (using Neo4j as an example in this embodiment), the following main types of nodes are created and maintained:

[0072] KnowledgePoint: The core entity with attributes such as kp_id (unique identifier), name, weight (dynamic weight), and description.

[0073] TextbookChapter (textbook chapter node): Stores compressed core textbook content, with attributes such as chapter_id, title, and compressed_content.

[0074] PastQuestion (Historical Question Node): Represents past exam questions, with attributes such as question_id, content, and difficulty.

[0075] PropositionRule (Proposition Specification Node): Stores the technical rules for propositions, and has attributes such as rule_id and rule_content (e.g., "Distractors should have realism").

[0076] CognitiveLevel (Cognitive Level Node): As an independent node, such as creating nodes named "Memory" or "Application", it facilitates hierarchical reasoning.

[0077] Simultaneously, the system defines and creates the following core relationship types to connect the aforementioned nodes, forming a knowledge semantic network:

[0078] BELONGS_TO: Connects the KnowledgePoint node and the TextbookChapter node, indicating that the knowledge point comes from a certain chapter of the textbook.

[0079] EXAMINES: Connects the PastQuestion node and the KnowledgePoint node, indicating that the question tests a specific knowledge point. This relationship may have a frequency (number of times tested) attribute.

[0080] REQUIRES: Connects the KnowledgePoint node and the CognitiveLevel node, representing the cognitive level required to master this knowledge point.

[0081] FOLLOWS: Connects the KnowledgePoint node and the PropositionRule node, indicating that propositions for this knowledge point should follow a certain rule.

[0082] PREREQUISITE: Connects two KnowledgePoint nodes, representing the prerequisite dependencies between knowledge points (e.g., "Knowledge Point A is a prerequisite for Knowledge Point B"). This relationship is the key point that the system needs to automatically infer and complete.

[0083] Implementation method for automatic knowledge graph relation completion:

[0084] To achieve automatic discovery and completion of PREREQUISITE relations, this embodiment employs a collaborative mechanism combining probabilistic reasoning and graph representation learning:

[0085] Feature Extraction and Probabilistic Inference (Bayesian Module): For any pair of knowledge points (A, B), the system extracts observational features from the graph, such as: whether they belong to the same chapter (co_chapter), whether they co-occur in past exam papers (co_occurrence), their order in the textbook table of contents (order), and the semantic similarity of their names (semantic_sim). Using a Bayesian probabilistic graphical model, these features are used as evidence E to calculate the posterior probability P(H_ab|E) of the hidden variable H_ab (i.e., "A is a priori knowledge of B" is true), which is then used as the first probability value.

[0086] Graph Representation Learning and Link Prediction (GNN Module): The system employs a Graph Neural Network (GNN) to learn node representations from the knowledge graph. Each node aggregates its neighbor information through multi-layer message passing, ultimately obtaining a vector representation containing graph structure information. Subsequently, a link prediction model (such as DistMult or a neural network) is used, taking the final vector representation of the knowledge point pair (A,B) as input, to calculate a second probability value indicating the existence of a PREREQUISITE relationship between them.

[0087] Probability Fusion and Relationship Creation: The system performs a weighted average of the first and second probability values ​​to obtain the comprehensive confidence score (Final_ab) indicating the existence of a prior relationship between the knowledge point pairs. A confidence threshold is set (e.g., 0.75). The system periodically scans all knowledge point pairs. If Final_ab exceeds the threshold and the PREREQUISITE relationship does not exist in the graph, a PREREQUISITE relationship edge from A to B is automatically created, and a confidence attribute is attached to this edge, recording the confidence score. The addition of the new relationship triggers the retraining of the GNN model, forming a self-reinforcing closed loop.

[0088] Implementation method for dynamic weight calculation of knowledge points:

[0089] The weight attribute of each KnowledgePoint node is dynamically calculated and updated using the following multi-factor weighting formula:

[0090] W_kp=α*F_freq+β*F_semantic+γ*F_cog

[0091] Wherein: Frequency factor (F_freq): Based on the relationship statistics between PastQuestion nodes and EXAMINES. Calculate the total number of times this knowledge point has been tested in past exam questions and normalize it. For example, F_freq = 1.5 for the top 10% of questions, 1.2 for the middle 80%, and 1.0 for the bottom 10%.

[0092] Semantic association factor (F_semantic): Using the Sentence-BERT model, the knowledge point name / description and the core main idea sentence of the exam syllabus are encoded into vectors respectively, and the cosine similarity is calculated. The similarity value is scaled and used as F_semantic.

[0093] Cognitive Level Factor (F_cog): Assigned a value based on the CognitiveLevel node of the knowledge point's REQUIRES. For example, F_cog = 1.0 for "Memory", 1.2 for "Understanding", 1.5 for "Application", and 1.8 for "Analysis and Creation".

[0094] Hyperparameters: α, β, γ are adjustable weight coefficients that satisfy α+β+γ=1. The initial values ​​can be set by domain experts (e.g., α=0.5, β=0.3, γ=0.2).

[0095] Once the system parses and inputs new past exam questions, it will automatically update the F_freq of the relevant knowledge points and recalculate its W_kp, thus realizing the dynamic evolution of the weights.

[0096] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0097] S201. Material Acquisition and Preprocessing.

[0098] The system collects original exam question texts from multiple sources (such as authoritative news websites, academic paper databases, special report databases, and local textbook resource databases) through web crawlers or API interfaces. These materials may exist in formats such as PDF, Word, HTML, or plain text.

[0099] First, the system uses tools such as Apache Tika to extract and parse text in multiple formats. For PDFs or scanned documents containing mixed text and images, the PaddleOCR engine is used for optical character recognition. The recognized text is then merged with the main text to form complete plain text content.

[0100] Subsequently, the system performs intelligent text segmentation on the cleaned long text. To improve the accuracy of subsequent retrieval, the system adopts a semantic-based segmentation strategy, prioritizing the use of SemanticChunker or similar models. By calculating the similarity of embedding vectors between sentences, segmentation is performed at boundaries where semantics change significantly, ensuring that each text chunk remains semantically coherent. Simultaneously, the system sets the text chunk size (e.g., 512–1024 characters) and overlapping areas (e.g., 50–100 characters) to maintain contextual integrity and avoid information fragmentation.

[0101] Each text block is given structured metadata, including:

[0102] source: the source of the material;

[0103] title: The title of the article;

[0104] topic: Topic tags identified and labeled using topic models such as BERTopic;

[0105] Summary: A summary of this text block generated using a summarization model such as BART or T5.

[0106] Finally, the system outputs a series of structured material fragments with rich metadata for subsequent vectorization processing.

[0107] S202, Vectorization of materials.

[0108] The system employs pre-trained text embedding models trained on large-scale Chinese corpora that demonstrate excellent performance in semantic retrieval tasks, such as bge-large-zh, text2vec-large-chinese, or M3E. Each structured text fragment (containing its text content and key metadata, such as the title) is input into the model, which outputs a high-dimensional semantic vector (e.g., 1024 dimensions). This vector captures the deep semantic information of the text fragment and is used for subsequent similarity retrieval.

[0109] S203, Knowledge Point Pre-association and Storage.

[0110] To establish an initial connection between the materials and the knowledge point system, the system employs one or more of the following pre-association strategies to tag each material fragment with a knowledge point:

[0111] Metadata-based rule matching: If the material fragment has highly structured metadata (e.g., the file name contains "Chapter 3_Newton's Laws of Motion.pdf"), the system will directly associate it with the KnowledgePoint node named "Newton's Laws of Motion" in the knowledge graph through regular expressions or keyword matching, and record its kp_id.

[0112] Based on keywords and entity links: The system uses the TextRank algorithm to extract keywords from the source material and uses the NER (Named Entity Recognition) model to identify proper nouns or subject-specific terms (such as "photosynthesis"). Subsequently, through entity linking technology, these entities are linked to the corresponding knowledge point nodes in the knowledge graph to obtain their kp_id.

[0113] Zero-shot classifier-based approach: The system constructs the association problem as a zero-shot text classification task. Using the names and descriptions of all knowledge points in the knowledge graph as category labels, and employing a text classification model such as Deberta or a specially fine-tuned model, the semantic relevance probability between the material fragment and each knowledge point is calculated. Knowledge points with probabilities exceeding a preset threshold (e.g., 0.6) are selected, and their kp_ids are used as the associated objects.

[0114] After the association is completed, the system stores the vector representation of each material fragment, the original text content, basic metadata, and a list of pre-associated knowledge point identifiers (related_kp_ids). In this embodiment, a vector database such as Milvus or Chroma is used for storage. The vectors are stored in a dedicated vector index, while the text, metadata, and related_kp_ids are stored in an associated metadata table or set, linked by unique IDs. This constructs a material vector database that is initially associated with the knowledge point system and supports efficient semantic retrieval.

[0115] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0116] S301, Extraction of Propositional Constraint Information

[0117] When the system receives a question request targeting a specific knowledge point (e.g., "Newton's First Law"), it first uses the unique identifier (kp_id) of that knowledge point to query and reason within the constructed structured knowledge network (i.e., knowledge graph) to extract the structured constraint information needed to generate the question. The extracted content includes:

[0118] Cognitive Level: By querying the REQUIRES relationship between the knowledge point node and the CognitiveLevel node, the required cognitive level (such as "application") can be determined.

[0119] Difficulty requirement: Based on the dynamic weight of the knowledge point node, the average difficulty statistics of related historical test questions, and the difficulty coefficient mapping rules preset by experts, a target difficulty value is calculated (for example, the value range is 0.0~1.0).

[0120] Question format rules: By querying the FOLLOWS relationship between the knowledge point node and the PropositionRule node, the applicable question-setting rules can be obtained, such as: "Single-choice questions should contain 4 options, and the correct option is unique" and "Essay questions should contain clear scoring points".

[0121] In addition, detailed descriptions, core concepts, common misconceptions, and related preceding / following knowledge points of the knowledge point can be extracted as deep semantic constraints when generating test questions.

[0122] S302, Relevant Material Search.

[0123] Based on the target knowledge point, the system initiates a retrieval process in the material vector database to obtain original material fragments that are highly semantically related to the knowledge point. The retrieval process is as follows:

[0124] Constructing Enhanced Query Vectors: The system does not directly use knowledge point names as queries. Instead, it combines the name, description, cognitive level, and related propositional norms of the knowledge points to form a rich query text (e.g., "Understanding and application of 'Newton's First Law,' cognitive level is 'application,' requiring the design of multiple-choice questions with practical scenarios"). Using the same pre-trained text embedding model (such as BGE-M3) as in the material vectorization stage, this query text is encoded into a high-dimensional query vector.

[0125] Perform an approximate nearest neighbor search: Input the query vector into a vector database based on an HNSW index (such as FAISS or Milvus). The system performs an approximate nearest neighbor search and returns the top K (e.g., K=5) clips with the highest similarity to the query vector. These clips have already been segmented into semantically coherent text blocks during the preprocessing stage and are accompanied by metadata such as source and topic.

[0126] Results filtering and optimization: The system can further utilize the pre-associated knowledge point identifiers (related_kp_ids) in the metadata of the material fragments to filter out fragments that are directly related to the target knowledge points, sort them according to semantic similarity scores, and finally select the 1 to 3 most relevant material fragments for subsequent use.

[0127] S303, Generative prompt text construction.

[0128] The system uses pre-defined prompt templates to structurally combine proposition constraints, retrieved relevant material fragments, and built-in proposition task instructions to generate high-quality, generative prompt text. An example template is shown below:

[0129] You are a subject matter exam question setter. Please generate a test question strictly according to the following requirements:

[0130] [Propositional Constraints]

[0131] Knowledge Point: {Knowledge Point Name}

[0132] Cognitive levels: {cognitive levels}

[0133] Difficulty: {Difficulty Value}

[0134] Question type requirements: {Question type rules}

[0135] Core Concepts: {Knowledge Point Descriptions}

[0136] [Reference Material]

[0137] {Insert 1-3 source clips here, each with its source indicated}

[0138] [Generate Instructions]

[0139] Based on the above knowledge points and reference materials, please create a question of the specified cognitive level and difficulty. Ensure the question stem is scientifically accurate, the answer choices are well-designed, the distractors are plausible, and provide a standard answer and brief explanation.

[0140] The system automatically fills the outputs of S301 and S302 into the corresponding positions in the template to generate the final large model instructions.

[0141] S304. Batch generation of initial draft test questions

[0142] The system inputs the generated prompt text constructed above into the selected large language model (e.g., GPT-4, ChatGLM, or Wenxin Yiyan). To balance the standardization, innovation, and diversity of the test questions, the system will adjust the following key generation parameters of the large language model:

[0143] Temperature: Set to 0.8~1.2 to encourage the model to produce moderate creative output while adhering to constraints.

[0144] top_p: Set to 0.9 to control the diversity of generated content.

[0145] Maximum generation length: 512~1024 tokens depending on the question type.

[0146] The system can generate multiple draft questions in batches by repeatedly calling the system or generating multiple drafts at once (for example, generating 3-5 questions from different perspectives in one request). All draft questions are parsed and stored as structured data, including fields such as question stem, option list, correct answer, and explanation, and then enter the subsequent question filtering and evaluation pipeline.

[0147] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0148] S401, Question Input and Simulated Answering.

[0149] The system inputs the draft test questions (including complete question stems, options, preset answers, and explanations) generated in the preceding steps into a simulation evaluation module. This module deploys at least one, and typically multiple, large language models (LLMs) with varying parameter sizes as simulated test-taker agents. For example, the system can simultaneously invoke a lightweight model (e.g., 1-B parameters), a medium-sized model (e.g., 7-14B parameters), and a large or specialized model (e.g., 72B or more parameters). Upon receiving the test questions, each model simulates the cognitive process of a real test-taker, independently generating its solution to the question, including: selected answer options (or, for subjective questions, the answer text), and key reasoning chains or problem-solving steps. The system records the output of each agent model.

[0150] S402, Multi-dimensional quality scoring.

[0151] The system automatically analyzes the answering process and results of each simulated test-taking agent based on pre-defined quality dimensions covering educational measurement and test design expertise, and generates quantitative scores for the initial draft of the test question across each dimension. Core quality dimensions include:

[0152] Logical consistency: Analyze whether there are logical contradictions or ambiguities between the statement in the question stem, the content of each option, and the reasoning process of the simulated agent. The score is based on the results of the contradiction detection model and rule verification.

[0153] Distractor validity: Evaluating whether incorrect options (distractors) possess "realism". This is determined by analyzing the simulation agent's degree of "hesitation" on incorrect options (e.g., length of reasoning steps, confidence level expression) and whether it is selected. Distractors that are easily eliminated by multiple agents or never considered score low.

[0154] Answer uniqueness and certainty: Check whether all high-capability agents (such as large models) can consistently derive a unique and correct preset answer, and whether the reasoning is conclusive. If there are discrepancies or too many additional assumptions are required to arrive at the answer, the score is reduced.

[0155] Cognitive level fit: Compare whether the simulated problem-solving process actually triggered by the test questions (such as whether it only requires memorization and reproduction or requires multi-step analysis and application) matches the target cognitive level specified in the test constraints.

[0156] Language standardization and scientific rigor: Use rules and models to check the accuracy of terminology, clarity of expression, and the presence of scientific errors.

[0157] The scoring algorithm for each dimension may combine rule matching, textual implication judgment, sentiment / determinism analysis, and metrics based on agent behavior statistics.

[0158] S403, Calculation of Comprehensive Quality Score and Threshold Determination.

[0159] The system employs a dynamically weighted ensemble learning algorithm to fuse scores from various quality dimensions. The weights can be adaptively adjusted based on question type, knowledge domain, or historical review feedback. An example of the calculation formula is as follows:

[0160] Overall quality score = w1 * logical consistency score + w2 * distractor effectiveness score + ... + wn * language standardization score

[0161] The sum of the weight coefficients is 1, and important dimensions (such as logical consistency and answer uniqueness) are usually given higher weights.

[0162] After calculating the overall quality score of the test questions (usually normalized to 0-100), the system compares it with a preset admission threshold. This threshold can be dynamically set by the administrator based on the quality requirements of the question bank (e.g., set to 80 points). The judgment logic is as follows:

[0163] If the overall quality score is greater than or equal to the acceptance threshold, the initial draft of the test question is deemed to be a qualified test question, automatically enters the qualified test question bank, and its quality level can be marked.

[0164] If the overall quality score is lower than the acceptance threshold, the question is deemed unqualified. The system can route it to the manual review stage or reject it directly and record the reason for failure (such as "invalid distractor"). This feedback can be used to iteratively optimize question prompts or generation models.

[0165] S404, Output and Closed Loop.

[0166] Finally, the system outputs a structured set of all qualified test questions, completing the entire process of automated question generation. Simultaneously, data generated during the simulation evaluation process (such as the answer trajectories and scoring details of each model) is recorded in a log, which can be used for subsequent question quality analysis, model optimization, and iterative updates of the knowledge graph, forming a continuously improving intelligent question generation closed-loop system.

[0167] In some embodiments, the intelligent question-generating system simulating expert question generation may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the intelligent question-generating system simulating expert question generation may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1(Description) The function of intelligent question generation that simulates expert question generation.

[0168] In this embodiment, the intelligent question-setting system simulating expert question setting can be divided into multiple functional modules based on its functions, such as... Figure 2 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0169] The knowledge graph construction module is used to construct a structured knowledge network containing knowledge point systems and their attribute relationships based on multiple propositional data.

[0170] The material processing module is used to collect the proposition materials and represent them in vector form, and to establish a material vector database associated with the knowledge point system.

[0171] The question generation module is used to obtain proposition constraint information from the structured knowledge network based on the target knowledge points, and retrieve relevant materials from the material vector database; combine the proposition constraint information and the relevant materials into generative prompts, and input them into the large language model to generate the initial draft of the question;

[0172] The test question evaluation module is used to input the initial test question into at least one large language model that acts as a simulated answer agent, obtain its answer process and results for the test question; based on the answer process and results, evaluate the quality of the initial test question and output qualified test questions.

[0173] Figure 3 The intelligent question-setting method for simulating expert question-setting provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structures involved in the embodiments of this invention do not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0174] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0175] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.

[0176] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0177] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0178] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0179] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0180] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0181] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0182] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0184] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. An intelligent question-setting method simulating expert question setting, characterized in that, include: Based on multiple sets of data, a structured knowledge network containing a knowledge point system and its attribute relationships is constructed. Collect the materials for the questions and represent them in vector form, and establish a material vector database associated with the knowledge point system; Based on the target knowledge points, proposition constraint information is obtained from the structured knowledge network, and relevant materials are retrieved from the material vector database; the proposition constraint information and the relevant materials are combined into generative prompts, which are then input into the large language model to generate the initial draft of the test questions; The initial draft of the test questions is input into at least one large language model that acts as a simulated answer agent to obtain its answer process and results. Based on the solution process and results, evaluate the quality of the initial draft of the test questions and output the qualified test questions; Based on data from multiple propositional bases, a structured knowledge network is constructed, encompassing a knowledge point system and its attribute relationships, including: Obtain various data sources for setting exam questions, including at least the exam syllabus, electronic textbooks, and past exam papers. The examination syllabus is analyzed to extract knowledge points, cognitive levels, and ability requirements, and the weights of knowledge points are dynamically calculated based on semantic association. The electronic teaching materials are analyzed to extract the core content paragraphs corresponding to the knowledge points; The past exam questions were analyzed to construct structured data representing the relationship between questions, knowledge points, cognitive levels, and difficulty. Based on the extracted knowledge points, weights, core content paragraphs, and structured data, a knowledge graph containing nodes, attributes, and relationships is constructed and stored. Graph reasoning models are used to infer and complete the logical dependencies between knowledge points. Based on the extracted knowledge points, weights, core content paragraphs, and structured data, a knowledge graph containing nodes, attributes, and relationships is constructed and stored, including: Define multiple types of nodes, including at least: KnowledgePoint nodes representing knowledge points, TextbookChapter nodes representing textbook chapter content, PastQuestion nodes representing historical test questions, PropositionRule nodes representing question-setting rules, and CognitiveLevel nodes representing cognitive ability levels. Define the relationship types connecting the nodes, including at least: the BelongsTo relationship connecting knowledge points and textbook chapters, the Examines relationship connecting historical test questions and knowledge points, the Requires relationship connecting knowledge points and cognitive ability levels, the Follows relationship connecting knowledge points and question specifications, and the Prerequisite relationship used to represent the logical dependencies between knowledge points. Prerequisite relationships between knowledge points are inferred and completed using graph reasoning models; The KnowledgePoint node has a dynamic weight attribute, and its weight value is calculated based on multi-factor weighting. The factors include at least: a frequency factor obtained based on the statistical analysis of the historical test data, an association factor obtained based on semantic similarity calculation, and a hierarchical factor determined based on the cognitive ability level. The graph reasoning model is used to infer and complete the prerequisite relationships between knowledge points, including: Based on the existing nodes and relationships in the knowledge graph, at least one pair of observation features between knowledge points are extracted, and a probabilistic reasoning model is used to calculate the first probability value of the existence of a prior knowledge relationship between the knowledge point pairs. The knowledge graph is subjected to graph representation learning. The neighbor information of the nodes is aggregated through message passing to update their vector representation. Based on the updated node vector representation, a second probability value is calculated through a link prediction model to determine whether there is a prior knowledge relationship between the knowledge point pairs. By combining the first probability value and the second probability value, a comprehensive confidence level is obtained to indicate that there is a prior knowledge relationship between the knowledge point pairs. When the comprehensive confidence level exceeds a preset threshold and the relationship does not exist in the graph, a corresponding prior knowledge relationship is automatically created.

2. The method according to claim 1, characterized in that, Collect test-taking materials and represent them in vector form, and establish a material vector database associated with the knowledge point system, including: Original proposition material texts are collected from at least one source of material, and the original proposition material texts are subjected to topic identification and summary generation to obtain structured material fragments; The structured material fragments are encoded into fixed-dimensional vector representations using a pre-trained text embedding model. Based on at least one pre-association strategy, the preliminary association between the structured material fragment and one or more knowledge points in the knowledge point system is determined, and the knowledge point identifier representing the preliminary association is stored as metadata corresponding to the vector representation.

3. The method according to claim 1, characterized in that, Based on the target knowledge points, propositional constraint information is obtained from the structured knowledge network, and relevant materials are retrieved from the material vector database; The propositional constraints and related materials are combined into generative prompts, which are then input into a large language model to generate a draft of the test questions, including: Based on the target knowledge points, propositional constraint information is extracted from the structured knowledge network. The propositional constraint information includes at least the cognitive level, difficulty requirements, and applicable question type rules of the target knowledge points. Based on the target knowledge points, at least one semantically related material fragment is retrieved from the material vector database; The proposition constraint information, the at least one material fragment, and the preset proposition task instructions are formatted and combined to construct generative prompt text; The generative prompt text is input into a large language model. By adjusting the generation parameters of the large language model to control the output characteristics, initial drafts of test questions that conform to the propositional constraints are generated in batches.

4. The method according to claim 1, characterized in that, Based on the aforementioned solution process and results, the quality of the initial draft of the test questions is evaluated, and qualified test questions are output, including: Based on multiple preset quality dimensions, the solution process and results generated by the simulated answer agent are analyzed to generate scores for the initial draft of the test questions on each quality dimension. Based on the scores across each quality dimension, the overall quality score of the initial draft of the test questions is calculated. The overall quality score is compared with a preset acceptance threshold to determine whether the initial draft of the test questions meets the standards.

5. An intelligent question-setting system simulating expert question setting, characterized in that, include: The knowledge graph construction module is used to construct a structured knowledge network containing knowledge point systems and their attribute relationships based on multiple propositional data. The material processing module is used to collect the proposition materials and represent them in vector form, and to establish a material vector database associated with the knowledge point system. The question generation module is used to obtain proposition constraint information from the structured knowledge network based on the target knowledge points, and retrieve relevant materials from the material vector database; combine the proposition constraint information and the relevant materials into generative prompts, and input them into the large language model to generate the initial draft of the question; The test question evaluation module is used to input the initial draft of the test question into at least one large language model that acts as a simulated answer agent, and obtain its answer process and results for the test question; Based on the solution process and results, evaluate the quality of the initial draft of the test questions and output the qualified test questions; Based on data from multiple propositional bases, a structured knowledge network is constructed, encompassing a knowledge point system and its attribute relationships, including: Obtain various data sources for setting exam questions, including at least the exam syllabus, electronic textbooks, and past exam papers. The examination syllabus is analyzed to extract knowledge points, cognitive levels, and ability requirements, and the weights of knowledge points are dynamically calculated based on semantic association. The electronic teaching materials are analyzed to extract the core content paragraphs corresponding to the knowledge points; The past exam questions were analyzed to construct structured data representing the relationship between questions, knowledge points, cognitive levels, and difficulty. Based on the extracted knowledge points, weights, core content paragraphs, and structured data, a knowledge graph containing nodes, attributes, and relationships is constructed and stored. Graph reasoning models are used to infer and complete the logical dependencies between knowledge points. Based on the extracted knowledge points, weights, core content paragraphs, and structured data, a knowledge graph containing nodes, attributes, and relationships is constructed and stored, including: Define multiple types of nodes, including at least: KnowledgePoint nodes representing knowledge points, TextbookChapter nodes representing textbook chapter content, PastQuestion nodes representing historical test questions, PropositionRule nodes representing question-setting rules, and CognitiveLevel nodes representing cognitive ability levels. Define the relationship types connecting the nodes, including at least: the BelongsTo relationship connecting knowledge points and textbook chapters, the Examines relationship connecting historical test questions and knowledge points, the Requires relationship connecting knowledge points and cognitive ability levels, the Follows relationship connecting knowledge points and question specifications, and the Prerequisite relationship used to represent the logical dependencies between knowledge points. Prerequisite relationships between knowledge points are inferred and completed using graph reasoning models; The KnowledgePoint node has a dynamic weight attribute, and its weight value is calculated based on multi-factor weighting. The factors include at least: a frequency factor obtained based on the statistical analysis of the historical test data, an association factor obtained based on semantic similarity calculation, and a hierarchical factor determined based on the cognitive ability level. The graph reasoning model is used to infer and complete the prerequisite relationships between knowledge points, including: Based on the existing nodes and relationships in the knowledge graph, at least one pair of observation features between knowledge points are extracted, and a probabilistic reasoning model is used to calculate the first probability value of the existence of a prior knowledge relationship between the knowledge point pairs. The knowledge graph is subjected to graph representation learning. The neighbor information of the nodes is aggregated through message passing to update their vector representation. Based on the updated node vector representation, a second probability value is calculated through a link prediction model to determine whether there is a prior knowledge relationship between the knowledge point pairs. By combining the first probability value and the second probability value, a comprehensive confidence level is obtained to indicate that there is a prior knowledge relationship between the knowledge point pairs. When the comprehensive confidence level exceeds a preset threshold and the relationship does not exist in the graph, a corresponding prior knowledge relationship is automatically created.

6. An intelligent question-setting device that simulates expert question setting, characterized in that, include: The memory is used to store the intelligent question-setting program that simulates expert questions; The processor, when executing the intelligent proposition program that simulates expert propositions, implements the steps of the intelligent proposition method that simulates expert propositions as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, The readable storage medium stores an intelligent question-setting program that simulates expert question setting. When the intelligent question-setting program that simulates expert question setting is executed by a processor, it implements the steps of the intelligent question-setting method that simulates expert question setting as described in any one of claims 1-4.

Images