A method for automatically evaluating difficulty of a question of fusing visual features and text semantics

Abstract

This invention discloses an automatic question difficulty assessment method that integrates visual features and textual semantics. Addressing the shortcomings of existing question difficulty assessment methods, which rely on manual annotation, are highly subjective, and are difficult to adapt to open-ended visual question answering, this invention constructs an unsupervised complexity modeling framework: After acquiring image-question pairs, a multi-visual language model is invoked to generate candidate answers; a semantic alignment network is used to calculate the semantic consistency score between the answer and the reference, simultaneously generating an Image Perception Quality Index (PQI); based on the performance of multiple models, the capability value is estimated and weighted to obtain the basic complexity; after PQI nonlinear correction and normalization, a continuous difficulty score is output, which can be mapped to a discrete difficulty level; simultaneously, a self-comparative evaluation mechanism is provided to achieve complexity ranking verification and calibration without manual labeling. This invention eliminates the need for manual difficulty annotation, takes into account the influence of textual semantics and image quality on difficulty, and provides a stable quantitative basis for adaptive learning, test paper generation, and model capability diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of multimodal artificial intelligence technology, specifically relating to an automatic problem difficulty assessment method that integrates visual features and text semantics. Background Technology

[0002] With the rapid development of online education, intelligent assessment, question bank operation, and multimodal AI, question difficulty assessment has become a core component of test paper generation, differentiated instruction, adaptive practice, data filtering, and model diagnostics. Image-based question-answering questions combine language comprehension and visual information reasoning, resulting in complex difficulty levels and necessitating stable and scalable assessment technologies.

[0003] The existing technology has obvious flaws: (1) Manual annotation / expert rules: high cost, long cycle, strong subjectivity, and difficulty in covering visual factors and open-ended answers; (2) Historical answer statistics: Relying on a large amount of data, difficult cold start, affected by group and environment interference, and inconsistent statistical standards for open-ended questions; (3) Single model / surface features: poor stability, unable to reflect the depth of reasoning, prone to high confidence errors, and difficult to integrate textual and visual dimensions.

[0004] To this end, this invention proposes an unsupervised, automated difficulty assessment scheme that takes into account both text semantics and image quality. Summary of the Invention

[0005] To overcome the problems in the background art, the present invention provides an automatic problem difficulty assessment method that integrates visual features and textual semantics.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: An automatic question difficulty assessment method integrating visual features and textual semantics includes the following steps: S1 Question Data Acquisition and Preprocessing: Acquire a sample set of questions to be evaluated. Each sample includes at least one image and the corresponding question text; if reference answers / standard answers are also stored in the question bank, they are also read. Images are processed to standardize size, color space, or through compression / decoding; question texts are standardized (e.g., removing meaningless whitespace, standardizing punctuation and encoding). The output is a "normalized image-question pair" and an optional "reference answer".

[0007] S2. Establish a set of multiple answering models: First, configure several visual language models for answering to form a set of answering models. This set can come from different models, or different versions / different inference configurations of the same model (e.g., different parameter scales, different inference strategies, or different prompt templates) to ensure that there are "ability differences" within the set; set a unified input and output format for each model (e.g., constrain the answer language, length, and whether uncertain answers are allowed). The output is a list of callable models and their interface configurations.

[0008] S3 Multi-model answer sampling: For each question sample, the image and question are input into each model in step S2 according to a uniform format to obtain the candidate answer generated by that model. To reduce randomness, the same model can be generated multiple times and deduplicated, or the most stable / consistent answer can be selected as the final candidate answer for that model. The output is "a set of multiple candidate answers for each question", and retains the correspondence between "candidate answers and the model identifier that generated them".

[0009] S4 Answer Quality Assessment: Establish an answer quality assessment module to determine whether a candidate answer is semantically consistent with the reference answer. When a reference answer exists: input the candidate answer and the reference answer into this module, and output a quality score (e.g., a continuous score from completely inconsistent to highly consistent), with optional consistency / inconsistency labels. When no reference answer exists: employ an alternative reference strategy. For example: (1) Use the "most models convergent answer" as a temporary reference; (2) Use a special judge model to judge the correctness of candidate answers; (3) Combine the consistency and mutual verification results between multiple models to form a quality score; (4) The semantic consistency calculation is a meaning-level matching, not a word-by-word comparison, and outputs a continuous score of 0~1.

[0010] The output is "the answer quality score for each question and each model".

[0011] S5 Model Capability Estimation: The overall performance of each model on a set of questions is statistically analyzed to form "model capability weights." Capability estimation can be derived from: the model's average answer quality on the question set, stability (consistency across multiple generation iterations), performance on high-quality image subsets, etc. Capability weights follow the principle that stronger models have higher weights; a stronger model performing poorly better indicates the difficulty of the question. The stronger the model, the higher its weight. These weights are used in subsequent difficulty fusion, reflecting the principle that "even a strong model performing poorly indicates a more difficult question." The output is "capability weights for each model."

[0012] S6 Basic Difficulty Fusion: For each question, the answer quality scores on different models are combined and weighted with the ability weights obtained in step S5 to form a "solvability / soothability" index: If most models (especially the more capable models) can produce high-quality answers, then the question is judged to be easier. If the quality of the model's answers is generally low, even strong models perform poorly, or there is a large discrepancy between models, then the question is judged to be more difficult. The output is a "basic difficulty indicator value for each question" (without considering corrections for image quality factors).

[0013] S7 Image Quality Assessment: For each question's image, the image quality assessment module is invoked to output a quality index reflecting the image's recognizability. This index comprehensively considers factors such as sharpness, noise, compression artifacts, occlusion, insufficient lighting, and insufficient resolution. The output is "the image quality index corresponding to each question".

[0014] S8 Difficulty Correction and Normalization: The basic difficulty in step S6 is corrected based on the image quality index in step S7. The correction rule is that the worse the image quality, the greater the increase in difficulty value; the better the image quality, the smaller the increase. Then, the corrected difficulty is normalized to ensure that the difficulty of different batches and different question banks falls within a uniform scale range, which is convenient for horizontal comparison and ranking. The output is the "final difficulty score".

[0015] S9 Difficulty Level Output: Maps the final difficulty score to discrete levels (e.g., several levels: easy / medium / hard, or 1 to several levels) according to a preset threshold, and outputs the question ID, difficulty score, difficulty level, and optional explanation information (e.g., percentage of strong model failures, degree of model divergence, correction magnitude due to low image quality, etc.) for question bank stratification, test paper generation, adaptive recommendation, and data filtering.

[0016] S10 Self-Comparison and Self-Check Calibration: When there is a lack of human-generated true difficulty values, a self-checking mechanism is introduced: sample question pairs for comparison to check whether "questions judged to be easier are indeed easier for the model to answer well"; if the self-check finds a large number of samples that violate this trend, it can be used to trigger parameter adjustments, remove abnormal questions, or re-estimate the model's ability weights, thereby improving the stability and credibility of the overall difficulty ranking; the self-comparison mechanism is: easier questions should show better model performance, and when the trend is violated, parameter adjustments or abnormal sample removal are triggered.

[0017] Preferably, the final output includes a continuous difficulty score, a discrete level, and explanatory information. The explanatory information includes the failure rate of strong models, the model divergence degree, and the magnitude of image correction. Each question has a continuous difficulty score (e.g., 0~100) and / or a discrete level (e.g., easy / medium / difficult). The key point is that the difficulty is not subjectively labeled by humans, but is automatically synthesized from three types of signals: "performance of multi-level respondents + consistency of answer meaning + degree of image recognizability".

[0018] The beneficial effects of this invention are: (1) Cost reduction and efficiency improvement: No need for manual annotation and a large amount of answer data, supports cold start and massive question bank; (2) Objective and stable: The standards are reproducible and the scale is uniform, avoiding subjective human bias; (3) Adaptation to open-ended questions: semantic consistency matching, compatible with synonymous expressions and multiple reasonable answers; (4) Taking visual factors into account: explicitly taking into account the cognitive load caused by image quality, and conforming to the real problem-solving scenario; (5) Robust and transferable: Multi-model weighting reduces the bias of single models, and the model update still maintains dimensional stability; (6) Unsupervised self-verification: Automatically verifies the reliability of results and supports quality control and parameter optimization; (7) High engineering value: Supports continuous and discrete levels, and is suitable for all scenarios of test paper assembly, adaptive learning, and model diagnosis. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a schematic diagram of the multi-model weighted fusion and difficulty correction of the present invention; Figure 3 This is a schematic diagram of the unsupervised self-comparison evaluation mechanism of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1 This embodiment is a visual question-and-answer exercise for elementary school mathematics geometry (with reference answers and high-definition images). Specifically, it is based on a K12 elementary school mathematics intelligent question bank, and the evaluation target is fourth-grade elementary school geometry recognition questions. It belongs to a closed-ended visual question-and-answer scenario with high-definition images and standard answers, and fully executes all the steps of this invention.

[0023] Question to be evaluated: The image is a high-resolution printed geometric figure combining an isosceles trapezoid and a triangle, without blurring or obstruction; the question text is "Please calculate what fraction of the total area the triangle part of the combined figure occupies"; the reference answer is "one-third".

[0024] Step S1 Data Preprocessing: Scale the geometric image to 512×512 pixels, convert it to RGB color space, and remove blank spaces from the image edges; remove extra spaces from the question text, standardize punctuation, and output a normalized image-question pair and reference answer.

[0025] Step S2: Construct a multi-model set: Select four visual language models with different capabilities, namely the strong model VLM-1 with 10B parameters, the medium-strong model VLM-2 with 7B parameters, the medium model VLM-3 with 3B parameters, and the weak model VLM-4 with 1B parameters. The input format is "image + question", and the output constraint is a concise answer of "number + score".

[0026] Step S3 Multi-model answering: Input the question into 4 models, each model generates an answer once, and after deduplication, the candidate answers are obtained: VLM-1 outputs "one-third", VLM-2 outputs "1 / 3", VLM-3 outputs "one-quarter", and VLM-4 outputs "one-half".

[0027] Step S4 Answer quality assessment: Based on the reference answer "one-third", the semantic consistency score is calculated through the semantic alignment network. VLM-1 is 1.0, VLM-2 is 0.98, VLM-3 is 0.1, and VLM-4 is 0.05. The scores are all continuous values ​​from 0 to 1. Non-word matching only judges semantic consistency.

[0028] Step S5: Model capability estimation: The average performance was statistically analyzed on 100 elementary school math geometry problems. The capability weights were 0.4 for VLM-1, 0.3 for VLM-2, 0.2 for VLM-3, and 0.1 for VLM-4. The stronger model had a higher weight, which is consistent with the principle that "the answer results of the stronger model are more referential".

[0029] Step S6 Basic Difficulty Integration: Weighted calculation of basic solvability index = 1.0×0.4+0.98×0.3+0.1×0.2+0.05×0.1=0.4+0.294+0.02+0.005=0.719, which is converted into a basic difficulty score of 28.1 (the higher the solvability, the lower the difficulty).

[0030] Step S7 Image Quality Assessment: The image is high-definition and flawless, with sharpness, contrast, and resolution all meeting the standards, generating a Perceived Quality Index (PQI) of 0.96.

[0031] Step S8 Difficulty Correction and Normalization: Due to the excellent image quality, the difficulty is only slightly adjusted. After correction, the difficulty score is 28.1 × 0.98 = 27.5. After normalization to the range of 0 to 100, the final difficulty score is 27.5.

[0032] Step S9: Level Mapping: Based on the threshold (0~30 Easy, 31~60 Medium, 61~100 Difficult), determine the level as Easy, output the question ID, difficulty score 27.5, level "Easy", and the explanation information is "Strong model all correct, small model discrepancies, excellent image quality".

[0033] Step S10 Self-verification: Extract this question and a medium-difficulty question with a difficulty score of 45 to form a question pair, verify that "the easy question model performs better", and the consistency meets the standard. No parameter adjustment is required.

[0034] This embodiment verifies the evaluation accuracy of the present invention in scenarios with high-definition images, closed-ended questions and answers, and reference answers. The evaluation results are completely consistent with the annotations by human experts. Objective difficulty values ​​can be quickly output without human intervention, which is suitable for the batch annotation needs of primary school math question banks.

[0035] Example 2 This embodiment is an open-ended question-and-answer exercise for junior high school science experiments (without reference answers + ordinary images). It is specifically applied to a junior high school science adaptive learning system. The evaluation object is an experimental phenomenon description question, which belongs to the open-ended visual question-and-answer, standard image, and no reference answer scenario. The focus is on verifying the semantic evaluation and self-verification capabilities of this invention under no-reference conditions.

[0036] Question to be evaluated: The image is a real photo of "iron wire burning in oxygen" in the laboratory. The image is not blurry but has slight reflections; the question text is "Please describe the experimental phenomenon in the picture in detail and explain the basic type of the reaction"; there is no preset reference answer.

[0037] Step S1 Data Preprocessing: Adjust the brightness of the experimental images to remove reflection interference and standardize the size; output the image after text normalization - Question correct, no reference answer.

[0038] Step S2: Construct a multi-model set: Select 5 visual language models with different inference configurations, including strong, medium and weak capabilities, and unify the output constraint as a complete sentence of "phenomenon description + reaction type".

[0039] Step S3: Multi-model answering: Five models generate candidate answers, among which the answers of three strong / medium models are similar: "The iron wire burns violently, sparks fly, and a black solid is produced, which is a combination reaction"; the answers of two weak models are different: "The iron wire burns and emits light, producing gas, which is a decomposition reaction".

[0040] Step S4 Answer quality assessment: There is no reference answer. A majority consensus + model cross-validation strategy is adopted. The convergent answers of the three models are used as temporary references. Semantic consistency scores: strong model 0.95, medium model 0.92, weak model 0.1, 0.08.

[0041] Step S5 Model capability estimation: Based on the average answer quality of 200 science experiment questions, capability weights are assigned: strong model 0.35, medium model 0.3, weak model 0.2, 0.15.

[0042] Step S6 Basic Difficulty Integration: Weighted calculation of basic solvability index = 0.95×0.35+0.92×0.3+0.1×0.2+0.08×0.15=0.3325+0.276+0.02+0.012=0.6405, basic difficulty score 35.95.

[0043] Step S7 Image quality assessment: The image has slight reflections and a PQI index of 0.78, which is considered a medium quality image.

[0044] Step S8 Difficulty Correction and Normalization: The image quality is average, so the difficulty is slightly increased. After correction, the difficulty score is 35.95 × 1.05 = 37.75. After normalization, the final score is 37.8.

[0045] Step S9: Level Mapping: Determined to be of medium level. The output includes the difficulty score, level, and explanatory information such as "moderate model divergence, medium image quality".

[0046] Step S10 Self-verification: Without manual difficulty labeling, 10 sets of questions were selected for self-comparison verification. 9 sets met the trend of "easier questions perform better", 1 abnormal sample was removed, the overall consistency met the standard, and automatic calibration was completed.

[0047] This embodiment verifies the adaptability of the present invention in open-ended question-and-answer scenarios without reference answers. By replacing the reference answer with majority consensus, the semantic consistency of open-ended answers is accurately quantified, solving the pain point that traditional methods cannot assess the difficulty of open-ended questions. At the same time, the self-verification mechanism ensures the reliability of the assessment results.

[0048] Example 3 This embodiment is a complex optical path diagram reasoning question and answer system for high school physics (with reference answers and low-quality images). It is applied to a hierarchical test paper generation system for high school physics question banks. The evaluation object is a complex optical reasoning question, which belongs to the closed-ended reasoning question and answer, low-quality blurry image, and scenario with reference answers. The focus is on verifying the core innovations of the present invention: image quality correction and strong model weighting.

[0049] The question to be evaluated is: The image is a light path diagram of a combination of convex and concave lenses taken with a mobile phone. It has problems such as blurring, occlusion, and loss of detail. The question text is: "Based on the light path in the figure, determine the imaging properties of this lens combination and calculate the focal length ratio". The reference answer is: "It forms an inverted and reduced real image with a focal length ratio of 2:1".

[0050] Step S1 Data Preprocessing: Perform basic noise reduction on the blurred image, normalize the text, preserve the original image quality features, and output the image-question pair and reference answer.

[0051] Step S2: Construct a multi-model set: Select 6 visual language models with different capabilities, including 2 strong models, 2 medium models, and 2 weak models, covering different parameter scales and inference capabilities.

[0052] Step S3: Multi-model response: Strong models can only make partial inferences due to image blurring, resulting in biased answers; medium models have a high error rate; weak models answer completely incorrectly, with extremely low semantic consistency between candidate answers and reference answers.

[0053] Step S4: Answer quality assessment: Calculate semantic consistency scores based on reference answers. Strong models score 0.3 and 0.25, medium models score 0.15 and 0.1, and weak models score 0.05 and 0.02. The scores objectively reflect the quality of the model's responses.

[0054] Step S5 Model capability estimation: Statistical analysis on 300 high school physics optics questions showed that the strong model had a weight of 0.25, 0.25, the medium model had a weight of 0.15, 0.15, and the weak model had a weight of 0.1, 0.1. The strong model had the highest weight percentage.

[0055] Step S6: Basic Difficulty Integration: Weighted Calculation of Basic Solvability Index = 0.3×0.25 + 0.25×0.25 + 0.15×0.15 + 0.1×0.15 + 0.05×0.1 + 0.02×0.1 = 0.075 + 0.0625 + 0.0225 + 0.015 + 0.005 + 0.002 = 0.182, basic difficulty score is 81.8.

[0056] Step S7 Image quality assessment: The image is blurry, occluded, and has insufficient resolution. The PQI perceptual quality index is only 0.32, which is considered a low-quality image.

[0057] Step S8 Difficulty Correction and Normalization: The image quality is extremely poor. The difficulty is significantly increased according to the non-linear correction rule. The corrected difficulty score is 81.8 × 1.2 = 98.16. The final score after normalization is 98.2.

[0058] Step S9 Level Mapping: Determine the difficulty level based on the threshold. The output includes the difficulty score, level, and explanatory information such as "strong model answers poorly, model divergence is extremely large, and extremely low image quality leads to a significant increase in difficulty".

[0059] Step S10 Self-verification: Pair this question with an easy question with a difficulty score of 30, and verify that "the difficult question model performs extremely poorly and the easy question model performs excellently". If the consistency is 100%, no calibration is required.

[0060] This embodiment fully verifies the innovative value of the image perception quality (PQI) correction of the present invention, taking into account the cognitive load caused by image blurring and occlusion into the difficulty, which is consistent with real problem-solving scenarios; at the same time, the multi-model weighted fusion avoids the bias of single models, accurately identifies the difficult problems of "poor image quality + high reasoning difficulty", and provides an accurate basis for the selection of high school physics difficult problems and the stratified test paper composition.

[0061] Example 4 This example is a visual Q&A exercise for vocational college electromechanical training (no reference answers + low-quality images).

[0062] See Figure 2 The diagram shows a multi-model weighted fusion and difficulty correction method for automatically assessing the difficulty of problems by fusing visual features and textual semantics.

[0063] Overall content of the illustration: Figure 2 is a visual representation of the core technical aspects of this invention, focusing on the key process of "multi-model answering - weighted fusion - visual feature correction". It is a detailed expansion of the core module of the processing layer in Figure 1. The core logic from image-question pair input to final difficulty score + level output is presented with unidirectional arrows, highlighting the two major innovative points of the invention: "multi-model ability difference" and "visual feature (PQI) fusion".

[0064] Input identifier: The image-question pair is the unique initial input. The downward-branching arrow points to model modules of different ability levels, indicating that the same question data is synchronously input into all models for parallel answering. Model Modules: Model A (high ability), Model B (medium ability), and Model C (low ability) are used to identify the ability differences of multiple model sets. This is a visual representation of the principle of "stronger models have higher weights" in this invention. The arrows of each model module point downwards to their corresponding candidate answer identifiers, representing that different ability models generate independent candidate answers. Semantic scoring and weighted fusion: The arrows of all candidate answers point to the semantic consistency scoring module, indicating that all candidate answers are uniformly evaluated for semantic quality; the semantic consistency score and the model ability level label point to the ability weighted fusion module together through arrows, indicating that the scoring results are fused and calculated in combination with the model ability weights; the ability weighted fusion arrow points downward to the basic difficulty value, indicating that the basic difficulty result generated after weighted fusion does not consider visual features; Visual Feature Correction: The image-problem pair points to the Image Quality Index (PQI) module via branch arrows, representing the extraction of visual quality features from the original image and the calculation of the PQI; the Image Quality Index (PQI) and the base difficulty value are connected by double arrows to the final calculation stage, representing the PQI as a correction factor for non-linear correction of the base difficulty value; Output identifier: The corrected result is indicated by an arrow pointing to the final difficulty score + level. This is the only output of this core process, and it will also serve as the input data for the output and verification layer in Figure 1.

[0065] The specific implementation process of this embodiment: This embodiment is a visual question-and-answer exercise for vocational college electromechanical training. It is an open-ended question-and-answer exercise (without fixed answers, but with limited operational points), low-quality images, and no reference answers. It is applied to the intelligent assessment system for vocational college electromechanical training. The evaluation process focuses on the core technical aspects shown in Figure 2, verifying the implementation effect of the core innovation of this invention in the scenario of no reference answers and low-quality images.

[0066] To be evaluated: The image is a photo taken at a vocational college electromechanical motor disassembly training site. It is severely blurry, some parts are obscured, and the lighting is dim (low quality). The question text is "Describe the key operation steps and precautions for motor disassembly based on the picture. List at least 3 steps + 2 precautions". There is no preset reference answer (open-ended answer, only the number of core points is limited).

[0067] S1 Problem Data Acquisition and Preprocessing: Perform basic noise reduction on low-quality images (preserving original blur and occlusion features without changing the essential image quality), and uniformly scale them to 512×512 pixels; remove meaningless whitespace from the problem text, standardize punctuation, and output normalized image-problem pairs as the initial input data for Figure 2.

[0068] S2: Establish a set of multi-answer model sets: A set of visual language models with different capabilities was constructed, including 3 high-capability models (12B, 10B, and 8B parameters), 2 medium-capability models (5B and 3B parameters), and 1 low-capability model (1.5B parameters). The input format was uniformly "image + electromechanical professional problem text", and the output constraint was "≥3 decomposition steps + ≥2 precautions", matching the capability level division of model A (high), B (medium), and C (low) in Figure 2.

[0069] S3 Multi-Model Response Sampling (corresponding to Figure 2, Model Module → Candidate Answer): The normalized image-question pairs were simultaneously input into six models, each generating an answer once. The correspondence between model identifiers and candidate answers was preserved. The performance of the answers showed significant differences in ability levels, as shown in Figure 2. High-capability model (A1 / A2 / A3): Can identify core disassembly parts, list 3-4 steps + 2-3 precautions, with only minor deviations in the order of a few steps; Medium-level capability model (B1 / B2): Can only identify 2 core steps, and there is 1 professional error in the precautions; Low-capability model (C1): Unable to identify disassembled parts; steps and precautions do not match the actual training operation.

[0070] S4 Answer Semantic Quality Score (corresponding to Appendix 2 Candidate Answer → Semantic Consistency Score): Since there was no reference answer, the alternative reference strategy of majority consensus + judge model of this invention was adopted: the convergent answers of 3 high-capability models were used as temporary references, and at the same time, 1 judge model in the field of electromechanical engineering was called to judge the correctness of key points. Finally, a semantic consistency score of 0 to 1 was calculated (only the professional consistency of the key points of operation was judged, and the exact same expression was not required). The result is as follows: A1 (0.90), A2 (0.88), A3 (0.85), B1 (0.45), B2 (0.40), C1 (0.10).

[0071] S5 Model Capability Weighting (corresponding to Appendix 2: Semantic Consistency Score + Model Capability → Weighted Fusion) On 100 visual question-and-answer questions in vocational electromechanical training, the average answer quality and accuracy of professional point recognition for each model were statistically analyzed. Ability weights were generated according to the principle of giving higher weights to stronger models, and matched with the weight gradients of high / medium / low ability models as shown in Appendix 2. The results are as follows: A1 (0.22), A2 (0.20), A3 (0.18), B1 (0.15), B2 (0.12), C1 (0.13), with a total weight of 1.

[0072] S6 Basic Difficulty Calculation (corresponding to Appendix 2: Weighted Fusion → Basic Difficulty Value): The solvability index is calculated by weighting the semantic consistency score and the model capability weights. This solvability index is then converted into a basic difficulty value using the formula: "Basic Difficulty Score = 100 - Solvability Index × 100". The calculation process is as follows: Ease of Solvability Index = 0.90×0.22 + 0.88×0.20 + 0.85×0.18 + 0.45×0.15 + 0.40×0.12 + 0.10×0.13 =0.198+0.176+0.153+0.0675+0.048+0.013=0.6555 Basic difficulty score = 100 - 0.6555 × 100 = 34.45.

[0073] S7 Image Quality Index (PQI) Assessment (corresponding to Appendix 2: Image-Problem Pair → PQI Indicators): The image quality assessment module is invoked to calculate the Perceived Quality Index (PQI) (0~1) by comprehensively considering four core indicators: image blur, occlusion rate, illumination, and part detail recognition. In this example, the image is of low quality, and the final PQI is 0.45.

[0074] S8 Difficulty Correction and Normalization (corresponding to Appendix 2: Base Difficulty Value + PQI → Final Result): Nonlinear correction is performed based on PQI, with the correction rule being "when PQI∈[0.4,0.5), the difficulty correction coefficient = 1.25" (for low-quality images, the difficulty is significantly increased to match the cognitive load of visual recognition in real problem-solving). The corrected scores are then normalized to the 0~100 range. The calculation process is as follows: The corrected difficulty score is 34.45 × 1.25 = 43.06. The final difficulty score after normalization is 43.1 (rounded to one decimal place).

[0075] S9 Difficulty Level Output (corresponding to Appendix 2: Final Difficulty Score + Level): The discrete difficulty levels are mapped according to preset thresholds (0~30 = easy, 31~60 = medium, 61~100 = difficult), and professional domain explanation information is output at the same time. The final output result is: Difficulty score: 43.1, Difficulty level: medium. Explanation: Strong model failure rate 0% (score ≥ 0.7 is acceptable), model divergence 0.72 (average score of high-capability model - score of low-capability model), image correction magnitude + 25% (correction coefficient - 1).

[0076] S10 Self-Comparison Automatic Calibration: This question was paired with an easy question from the electromechanical training diagram with a difficulty score of 28. The self-comparison mechanism of this invention was used to verify the trend that "the easy question model performs better". The results showed that the average semantic score of the easy question model (0.81) was significantly higher than that of this question (0.62), and the consistency was met. At the same time, 15 similar question pairs were sampled for batch verification. Only one abnormal sample was removed, and the rest were in line with the trend, and automatic calibration was completed.

[0077] Verification effect of this embodiment This embodiment focuses on the core technical aspects shown in Figure 2, perfectly implementing the two major innovations of "multi-model weighted fusion" and "PQI visual feature correction." In a vocational college professional training scenario with no reference answers and low-quality images, the semantic quality assessment of open-ended questions and answers is achieved through "majority consensus + referee model." At the same time, the PQI nonlinear correction of low-quality images fully takes into account the visual cognitive load. The assessment results are completely consistent with the "medium" level manually marked by experts in electromechanical professional training, verifying the effectiveness and robustness of the core technical aspects of this invention and adapting to the visual question-answering difficulty assessment needs in the vocational education professional field.

[0078] Example 5 This example is a visual Q&A session on a high school chemistry experimental setup (with reference answers and medium-resolution images).

[0079] SeeFigure 3 The diagram shows an unsupervised self-comparison evaluation mechanism for an automatic problem difficulty assessment method that integrates visual features and textual semantics.

[0080] Figure 3 is a visual representation of the core unsupervised self-verification process of this invention, focusing on the logic of "consistency verification of difficulty ranking without manual difficulty labeling". This is the key module that distinguishes this invention from traditional manual labeling and enables unsupervised quality control. It follows the self-comparison automatic calibration process of the output and verification layer in Figure 1, and presents the complete verification process from the selection of question pairs to the stable difficulty ranking with a unidirectional arrow, highlighting the core verification principle that "the easier question model performs better".

[0081] Appendix Figure 3 The question identifiers to be verified in the text are: Question X (easy) and Question Y (difficult) as the core identifiers, representing question pairs of different difficulties pre-judged from the question pool that has been evaluated. They are the basic inputs for self-comparison verification. The two are side by side and point downwards with a converging arrow to the core verification module, which means that the model's answer performance for the two questions is compared and analyzed simultaneously. Core Verification Module: Self-Comparison Consistency Test (Easy Questions Should Perform Well) is the core module of the entire diagram. It receives the input of question pairs and is the execution carrier of the verification rules. The core verification principle of this invention is embedded in the module—the model answer performance of easy questions should be significantly better than that of difficult questions, providing a unique standard for judging the verification results. Model performance indicators: The core verification module branches downwards into two arrows, pointing to high performance and low performance indicators respectively. Question X (easy) corresponds to high performance and question Y (difficult) corresponds to low performance, representing normal results that meet the verification principles and are the ideal output of self-comparison verification. If reverse performance occurs (low performance for easy questions and high performance for difficult questions), it is an abnormal result that does not meet the verification principles, triggering subsequent parameter adjustment / abnormal sample removal operations. Final output identifier: The confluence arrows of high performance and low performance ultimately point to the stable difficulty ranking identifier, indicating that the question pairs that have been verified and meet the principles have valid difficulty judgment results and can be included in the final question bank difficulty ranking system; if abnormal results occur, they will be pointed to the model capability weighting, basic difficulty calculation and other modules in the processing layer of Figure 1 through implicit feedback arrows (logical feedback in the diagram), triggering parameter readjustment or abnormal question removal, and then returning to this stage for re-verification until a stable difficulty ranking is achieved.

[0082] The specific implementation process of this embodiment: This embodiment is a visual question-and-answer format for high school chemistry experimental devices. It belongs to the closed-ended reasoning question-and-answer format (limited to the analysis conclusion of the device), with medium-quality images and reference answers. It is applied to a high school chemistry intelligent test paper generation system. The evaluation process focuses on the unsupervised self-comparison evaluation mechanism shown in Figure 3. The core verification of this invention is its ability to stabilize the difficulty ranking through self-comparison verification without human difficulty truth labels. At the same time, it covers all steps S1-S10 and forms a complete closed loop with the technical links in Figures 1 and 2.

[0083] The pool of questions to be evaluated includes two visual question-and-answer questions about experimental setups selected from a high school chemistry question bank as a self-comparison verification pair. After basic evaluation based on this invention, the pre-determined difficulty level is "Easy + Difficult". Specific information is as follows: Question X (Easy to predict): The image is a diagram of a potassium permanganate heating apparatus for laboratory oxygen production (medium quality, slight paper wrinkles, no missing details); the question text is "Point out two obvious errors in the experimental apparatus in the diagram"; the reference answer is "① The test tube opening is not plugged with cotton; ② The test tube opening is tilted upwards".

[0084] Question Y (Predicted Difficulty): The image is a diagram of a combined apparatus for chlorine preparation and tail gas treatment (medium quality, slightly blurred at the connection points, requiring multiple steps of reasoning); the question text is "Identify the function of the apparatus in the diagram and 3 errors, and explain the principle of tail gas treatment"; the reference answer is "① Saturated saline solution is connected to the separatory funnel to remove HCl from Cl2; Errors: ① The long-necked funnel is not liquid-sealed; ② The tail gas treatment does not use an anti-backflow device; ③ The alcohol lamp does not heat the round-bottom flask evenly; Tail gas treatment principle: NaOH solution undergoes a disproportionation reaction with Cl2, absorbing toxic gases to prevent pollution."

[0085] S1 Problem Data Acquisition and Preprocessing: The images of both questions were uniformly processed as follows: scaled to 512×512 pixels, edge blanking was removed, and the original features of the medium-quality images (wrinkles, slight blurring) were preserved. The question text and reference answers were standardized (extra spaces were removed, and chemical terminology was standardized). The standardized images of the two questions, along with the question pair and reference answer, were output as the raw data for subsequent evaluation.

[0086] S2: Establish a set of multi-answer model sets: Four visual language models (VLMs) with different capabilities in the chemical domain were selected to construct an ensemble, including two strong models (10B and 8B parameters, adapted for chemical professional reasoning), one medium model (5B parameters), and one weak model (2B parameters). The unified input format is "image + chemical question text", and the output constraints are standardized chemical professional terminology and clear answer points, providing a unified basis for multi-model answering and subsequent self-comparison and verification of model performance analysis.

[0087] S3 Multi-Model Response Sampling: The normalized data for questions X and Y were input into four models respectively. Each model generated an answer for each question and removed duplicates, while retaining the correspondence between model identifiers and candidate answers. The model's performance on the two questions showed a clear hierarchical difference (providing a basis for subsequent self-comparison and verification): Question X (Easy): Strong model answers all questions correctly, medium model answers one question correctly and makes one mistake, weak model answers one question correctly. Question Y (Difficult): The strong model answers only 2 errors and 1 function of the device correctly, the medium model answers 1 error correctly, and the weak model answers all questions incorrectly.

[0088] S4 Answer Semantic Quality Score: Based on the reference answers to the two questions, the semantic consistency score at the meaning level was calculated using a semantic alignment network in the chemistry domain (a continuous value of 0 to 1, where consistency between chemical terms and reasoning conclusions is considered valid, not a word-by-word matching).

[0089] S5 Model Capability Weighting: The average answer quality and professional reasoning accuracy of each model were statistically analyzed on 150 high school chemistry experiment setup questions. Ability weights were generated according to the principle that stronger models have higher weights (the total weights sum to 1). The results are shown in the table above, which provides a weighted basis for the calculation of basic difficulty.

[0090] S6 Basic Difficulty Calculation: For both questions, a weighted average of the semantic consistency score and the model capability weight is calculated. This average is then converted into a basic difficulty value using the formula: "Basic Difficulty Score = 100 - Solvability Index × 100" (higher solvability indicates lower difficulty). The calculation process is as follows: Question X (Easy): Ease of solving index = 1.00×0.40+0.99×0.30+0.50×0.20+0.50×0.10=0.4+0.297+0.1+0.05=0.847; Basic difficulty score = 100 - 0.847 × 100 = 15.3.

[0091] Question Y (Difficult): Ease of solving index = 0.65×0.40+0.62×0.30+0.25×0.20+0.05×0.10=0.26+0.186+0.05+0.005=0.501; The basic difficulty score is 100 - 0.501 × 100 = 49.9.

[0092] S7 Image Quality (PQI) Assessment: The image quality assessment module is called to calculate the Perceived Quality Index (PQI) (0~1) by combining sharpness, detail recognition, and degree of occlusion / wrinkling. Both questions are medium quality images, and the PQI value is the same: PQI=0.80.

[0093] S8 Difficulty Correction and Normalization: Nonlinear correction was performed based on PQI, with the correction rule being "when PQI∈[0.8,0.9), the difficulty correction coefficient = 1.04" (for medium-quality images, the difficulty was slightly increased). The corrected score was then normalized to the range of 0~100. The final score was calculated as follows: Question X (Easy): The corrected score is 15.3 × 1.04 = 15.91, and the normalized final score is 15.9.

[0094] Question Y (Difficult): The corrected score is 49.9 × 1.04 = 51.89, and the normalized final score is 51.9.

[0095] S9 Difficulty Level Output: The system maps discrete difficulty levels to preset thresholds (0~30 = Easy, 31~60 = Medium, 61~100 = Difficult), and outputs explanations for both questions. The basic assessment results are as follows: Question X: ID=HX2026032401, score 15.9, level easy; Explanation information: strong model failure rate 0%, model divergence 0.49, image correction magnitude +4%.

[0096] Problem Y: ID=HX2026032401, Score 51.9, Level Medium (Entry-level Difficulty); Explanation: Strong model failure rate 0%, Model divergence 0.57, Image correction magnitude +4%.

[0097] S10 Self-Comparison Automatic Calibration: This step is the core of Example 6, and it fully follows the unsupervised self-comparison evaluation mechanism shown in Figure 3. No manual difficulty labeling is involved; the core principle is that "the model performs better on easier questions." Consistency checks are performed on question pairs X (easy) + Y (difficult). The specific process is as follows: Question pair selection: According to the logic in Figure 3, randomly select question pairs pre-judged as "easy + difficult" from the chemistry question bank completed by the basic assessment (this question is X+Y). At the same time, select 5 sets each of "easy + medium" and "medium + difficult" question pairs to form a batch verification sample to ensure the stability of the ranking. Self-comparison consistency test (Figure 3, Core Module): Using the model's average semantic consistency score and the failure rate of strong models as the core indicators of model performance, the verification principle is applied—both indicators for easy questions should be significantly better than those for difficult questions. The performance indicators for the two questions are calculated as follows: Average model scores: Question X = (1.00 + 0.99 + 0.50 + 0.50) / 4 = 0.7475; Question Y = (0.65 + 0.62 + 0.25 + 0.05) / 4 = 0.3925; Strong model failure rate (score <0.8 is considered a failure): Question X=0%; Question Y=0% (strong model scores are all ≥0.6, not reaching the failure threshold). Test results: The average score of the model for question X is significantly higher than that for question Y, which fully conforms to the core principle of "easy questions should perform well" in Figure 3.

[0098] Abnormal result judgment and handling: The 12 sets of question pairs (including X+Y) were batch verified. Among them, 11 sets met the verification principle, and 1 set was an abnormal sample (the average score of the easy question model was lower than that of the difficult question). According to the logical feedback rules in Figure 3, the abnormal sample was removed (it was determined that the question image / text had defects and was not suitable for the evaluation method), and the model parameters were not adjusted. Stable difficulty ranking (see Figure 3, final output): The 11 pairs of questions that have been verified to meet the criteria have valid basic assessment difficulty scores and level determination results. Question X (15.9, easy) and Question Y (51.9, medium) are included in the final stable difficulty ranking system of the high school chemistry experiment device question bank, and self-comparison automatic calibration is completed.

[0099] The verification results of this embodiment: This embodiment precisely adapts to the unsupervised self-comparison evaluation mechanism shown in Figure 3. Without the presence of manual difficulty truth labels, it verifies the effectiveness of the self-verification step of this invention through the complete process of "question pair selection - consistency verification - anomaly handling - stable ranking". At the same time, this embodiment forms a complete closed loop with the technical steps in Figure 1 (full process) and Figure 2 (core fusion correction). In the high school chemistry visual question-answering scenario, the evaluation results are completely consistent with the difficulty ranking judgment of chemistry experts. Furthermore, the batch question pair verification eliminates abnormal samples, ensuring the stability and credibility of the question bank difficulty ranking.

[0100] Furthermore, this embodiment verifies the universality of the mechanism shown in Figure 3: it is not only applicable to "easy + difficult" question pairs, but also covers the comparison of the entire difficulty range of "easy + medium" and "medium + difficult". It provides a reproducible verification method for unsupervised difficulty assessment and ranking of massive question banks, solves the pain points of traditional assessment methods that rely on manual labeling and are easily affected by subjective factors in ranking, and perfectly adapts to the core requirement of accurate difficulty ranking for intelligent test paper generation in high school chemistry.

[0101] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for automatically assessing the difficulty of questions by integrating visual features and textual semantics, characterized in that, Includes the following steps: S1: Obtain and preprocess the sample questions to be evaluated. Each sample contains an image, the corresponding question text, and optional reference answers. Output normalized image-question pairs. S2: Construct a collection of multi-visual language models with different capabilities, and unify the input and output formats; S3: Input the same question into different models to generate multiple candidate answers and retain the model identifier; S4: Calculate the answer quality score for each candidate answer through semantic consistency. If there is no reference answer, use majority consensus or a referee model instead. S5: Estimate relative ability based on the overall response performance of the model and generate corresponding ability weights; S6: Weight the answer quality scores according to ability weights to obtain the basic difficulty indicator value of the question; S7: Evaluate image sharpness, noise, occlusion, lighting and other indicators to generate the Image Quality Index (PQI); S8: The PQI is used to perform non-linear correction on the basic difficulty and normalize it to a uniform range to obtain the final difficulty score; S9: Map the difficulty score to discrete difficulty levels and output them; S10: When there are no manual labels, the difficulty ranking consistency is checked by comparing the questions themselves, and automatic calibration is completed.

2. The method for automatically assessing the difficulty of a question by integrating visual features and textual semantics as described in claim 1, characterized in that, In S4, semantic consistency calculation is a meaning-level matching, not a word-by-word comparison, and outputs a continuous score of 0 to 1.

3. The method for automatically assessing the difficulty of a question by integrating visual features and textual semantics as described in claim 1, characterized in that, In S5, the ability weights follow the principle that stronger models have higher weights, and stronger models that perform poorly are better able to represent the difficulty of the questions.

4. The method for automatically assessing the difficulty of a question by integrating visual features and textual semantics as described in claim 1, characterized in that, The correction rule in S8 is that the worse the image quality, the greater the increase in the difficulty value; the better the image quality, the smaller the increase.

5. The method for automatically assessing the difficulty of a question by integrating visual features and textual semantics as described in claim 1, characterized in that, The S10 self-comparison mechanism is as follows: easy questions should present better model answer performance, and when the trend is violated, parameter adjustment or abnormal sample removal is triggered.

6. The method for automatically assessing the difficulty of a question by integrating visual features and textual semantics as described in claim 1, characterized in that, The final output includes a continuous difficulty score, a discrete level, and explanatory information. The explanatory information includes the failure rate of strong models, the model divergence degree, and the magnitude of image correction.

Images