Physicochemical and biological experiment report review feedback method and system based on multi-modal large model

Abstract

This invention relates to the field of artificial intelligence education technology, specifically to a method and system for reviewing and providing feedback on physics, chemistry, and biology lab reports based on a multimodal large-scale model. The method includes: acquiring lab report question data, student answer data, and scoring standard data; vectorizing the question data and retrieving scoring criteria from a scoring knowledge base; selecting either a text-based or image-based scoring process based on the answer type, and calling a fine-tuned multimodal large-scale language model for scoring reasoning; calling an error identification module and a teaching suggestion generation module based on the scoring results; summarizing the complete review results and calculating the joint confidence level; and determining whether to push the results for manual review based on the confidence level. This invention can improve the accuracy, fine-grained understanding, and quality of teaching feedback in lab report review, enabling the self-evolution of the review system.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for reviewing and providing feedback on physics, chemistry, and biology experiment reports based on a multimodal large model. Background Technology

[0002] With the deepening development of educational informatization, secondary school physics, chemistry, and biology experimental teaching has placed higher demands on efficient, objective, and quantifiable evaluation methods. Traditional experimental report grading mainly relies on teachers to complete it manually, which has problems such as long grading cycles, high labor costs, and difficulty in unifying grading standards. Especially when dealing with open-ended answers, it is easily affected by subjective judgment, affecting the fairness and consistency of the evaluation.

[0003] To improve grading efficiency, existing automated scoring systems mostly employ rule-based or statistical learning methods. For fill-in-the-blank questions, common techniques include keyword matching, string similarity calculation, or semantic comparison based on word vectors; for drawing questions, they often rely on template matching, edge detection, or specific symbol recognition algorithms from computer vision. These methods are effective for questions with high structure and fixed answer formats.

[0004] However, many questions in high school physics, chemistry, and biology lab reports are open-ended and unstructured. For example, fill-in-the-blank questions often require students to describe experimental phenomena, explain principles, or analyze errors, resulting in diverse and semantically complex answers. Drawing questions involve hand-drawn circuit diagrams, experimental apparatus diagrams, or biological structure diagrams, and students' drawing styles vary greatly, exhibiting issues such as jittery lines, distorted proportions, and non-standard symbols. Existing automatic scoring technologies struggle to accurately understand the semantic meaning of open-ended texts and cannot perform robust, fine-grained structural analysis of non-standard hand-drawn graphics, leading to low scoring accuracy and poor adaptability.

[0005] Furthermore, current systems generally only output scores or simple right / wrong judgments, lacking diagnosis of the reasons for errors and targeted learning suggestions, making it difficult to meet the teaching needs of formative assessment. At the same time, once the scoring criteria are fixed, updating and maintaining them is difficult, making it hard to adapt to changes in the teaching syllabus or the introduction of new question types. Although some systems support a certain degree of adaptation by updating the rule base or template library, the operation is complex, the granularity is coarse, and it cannot dynamically link the logical relationships between different knowledge points.

[0006] In summary, existing technologies have significant shortcomings in terms of scoring accuracy, feedback depth, system flexibility, and teaching support capabilities. There is an urgent need for an intelligent grading method that can effectively handle open-ended texts and non-standard hand-drawn images, support refined scoring, provide teaching-oriented feedback, and have a flexible knowledge update mechanism. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a method and system for reviewing and providing feedback on physics, chemistry, and biology lab reports based on a multimodal large model. This system aims to achieve high precision, interpretability, and evolvability in lab report review, thereby improving review accuracy and the quality of teaching feedback.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] On one hand, embodiments of the present invention provide a method for reviewing and providing feedback on physics, chemistry, and biology experiment reports based on a multimodal large model, the method comprising the following steps:

[0010] Acquire experimental report question data, student answer data, and scoring standard data, wherein the student answer data includes text answer data or image answer data;

[0011] The experimental report question data is input into a multimodal embedding model for vectorization processing to generate question embedding vectors. Based on the question embedding vectors, semantically similar knowledge records are retrieved from a preset scoring knowledge base as scoring basis data. The scoring knowledge base stores joint embedding vectors of structured scoring knowledge documents.

[0012] When the candidate's answer data is text answer data, the experimental report question data, the text answer data, the scoring basis data, and the scoring module system prompts are integrated into a text scoring instruction. The text scoring instruction is then input into a fine-tuned multimodal large language model for scoring reasoning, and a preliminary review result is output. The preliminary review result includes a score field and a scoring basis field.

[0013] When the candidate's answer data is image-based, the image-based answer data is input into a lightweight visual model for component detection, outputting component presence information and component annotation maps. Metadata is generated based on the component presence information and the scoring criteria data. The component annotation maps, the metadata, and the system prompts from the scoring module are integrated into an image scoring instruction. The image scoring instruction is input into the finely tuned multimodal large language model for scoring reasoning, and a preliminary evaluation result is output. The component presence information includes center coordinates, bounding boxes, and component types.

[0014] Based on the scoring criteria field and the candidate's answer data, the error identification module is called to extract error semantic fragments, and the error semantic fragments are classified and matched in combination with the error pattern knowledge base to output the error type field;

[0015] Based on the score field, the scoring basis field, and the error type field, the teaching suggestion generation module is invoked to generate a teaching suggestion field;

[0016] The score field, the scoring basis field, the error type field, and the teaching suggestion field are summarized to form a complete review result;

[0017] Calculate the joint confidence level based on the complete review results. When the joint confidence level is less than a preset threshold, mark the complete review results as pending manual review and push them to the teacher's end; otherwise, output the complete review results directly.

[0018] Optionally, before inputting the experimental report title data into the multimodal embedding model for vectorization processing, the method further includes:

[0019] For short answer and fill-in-the-blank questions and drawing questions in physics, chemistry and biology lab reports, a structured scoring standard is developed. The structured scoring standard includes question scores, reference answers and scoring details. The scoring standard for drawing questions is further subdivided into independent knowledge units, which include rules on the existence of components, rules on connection relationships, rules on drawing standardization, and rules on the accuracy of annotation.

[0020] Construct a structured scoring knowledge document. Each of the structured scoring knowledge documents includes a question text field, a reference answer field, a scoring criterion field, and a question score field. The reference answer field stores text content or image paths.

[0021] The structured scoring knowledge document is cleaned and preprocessed, the text content is segmented and stop word removed, and the image content corresponding to the image path is uniformly cropped to a preset pixel size.

[0022] The preprocessed structured scoring knowledge document is jointly vectorized using a domain-adaptive multimodal embedding model to generate a joint embedding vector. The joint embedding vector is then stored in a vector database to form a dynamically updatable scoring knowledge base.

[0023] Optionally, the step of employing a domain-adaptive multimodal embedding model to jointly vectorize the preprocessed structured scoring knowledge document to generate a joint embedding vector includes:

[0024] The title text field is input into the text encoder to generate a text embedding vector. The image content corresponding to the image path is input into the visual encoder to generate a visual embedding vector. The visual encoder adopts a visual Transformer architecture based on adapter fine-tuning.

[0025] The visual embedding vector and the text embedding vector are mapped to the same feature space through a cross-modal alignment module, which employs a learnable projection layer and a lightweight cross-attention layer.

[0026] The mapped visual embedding vector and the text embedding vector are fused to output a joint embedding vector, which is a fixed-dimensional vector that has been L2 normalized.

[0027] Optionally, before inputting the image answer data into the lightweight visual model for component detection, the method further includes:

[0028] A self-supervised pre-trained visual Transformer network was selected as the basic model to construct a training dataset of symbols for physics, chemistry and biology disciplines. The training dataset includes physical circuit diagrams, chemical device diagrams and biological structure diagrams.

[0029] A parallel adapter module is added to the base model. The adapter module adopts a fully convolutional structure. The output features of the adapter module are at the same scale as the output features of the attention module of the base model. The output features of the adapter module focus on the detailed information of the symbol pattern.

[0030] A gating unit is added before the output features of the adapter module are fused with the output features of the base model. The gating unit is used to control the activation state of the adapter module.

[0031] The context features of the base model and the detail features of the adapter module are subjected to bilinear pooling to enhance the correlation between features;

[0032] Freeze the backbone parameters of the base model, train only the parameters of the adapter module, and use a weighted combination of contrastive loss, bounding box regression loss and classification cross-entropy loss as the loss function to fine-tune the base model and obtain a lightweight visual model.

[0033] Optionally, calculating the joint confidence level based on the complete review results includes:

[0034] Obtain the historical manual confirmation rate of the knowledge unit corresponding to the scoring basis data, the historical average score rate of the questions corresponding to the experimental report question data, and the historical performance consistency score of the students corresponding to the candidate answer data;

[0035] The historical manual confirmation rate, the historical average score rate, and the historical performance consistency score are weighted and summed to generate a joint confidence score. The weight coefficients of the weighted summation are dynamically set according to the subject type.

[0036] Optionally, the method further includes:

[0037] Extract the candidate answer data marked as awaiting manual review, and the review result data after manual review and correction, as low-confidence sample data;

[0038] The low-confidence sample data is vectorized to generate sample embedding vectors;

[0039] A density clustering algorithm based on cosine distance is used to perform cluster analysis on the sample embedding vectors to identify approximate error clusters and new error clusters;

[0040] Candidate error patterns are generated based on clustering results. Each candidate error pattern includes an error summary, typical examples, and cause speculation. The candidate error patterns are then pushed to the teacher's end for review.

[0041] When the candidate error pattern passes the review, the candidate error pattern is automatically injected into the error pattern knowledge base and associated with the corresponding knowledge point.

[0042] Optionally, before inputting the text scoring instruction or the image scoring instruction into the fine-tuned multimodal large language model for scoring inference, the method further includes:

[0043] Construct a supervised fine-tuning dataset for experimental graphic scoring instructions. Each sample in the dataset contains the question text, the examinee's answer, the scoring criteria, and the standard review result.

[0044] The lightweight visual model is used as a visual tokenizer, and the visual tokenizer outputs a sequence of visual tokens.

[0045] The visual token sequence is processed by a projection layer and then concatenated with a text token. The result is then input into a text decoder. The projection layer employs a projection mechanism similar to a contrastive language image pre-training architecture.

[0046] The parameters of the visual word segmenter and the text decoder are frozen, and only the parameters of the projection layer are fine-tuned to complete the supervised fine-tuning of the multimodal large language model, resulting in the fine-tuned multimodal large language model.

[0047] Optionally, based on the scoring criteria field and the candidate's answer data, the error identification module is invoked to extract error semantic fragments, and the error semantic fragments are categorized and matched using an error pattern knowledge base to output an error type field, including:

[0048] The error identification module loads a dedicated system prompt word template, which guides the multimodal large language model to extract erroneous semantic fragments from the scoring criteria field that do not match the candidate's answer data.

[0049] The extracted error semantic fragments are matched with static rules and historical self-discovery patterns in the error pattern knowledge base to determine the error type label;

[0050] The output contains structured data including the error type field, which supports multi-label format.

[0051] On the other hand, embodiments of the present invention provide a physics, chemistry, and biology experiment report review and feedback system based on a multimodal large model, including:

[0052] At least one processor;

[0053] At least one memory for storing at least one program;

[0054] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0055] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0056] The beneficial effects of this invention are as follows: This invention discloses a method and system for reviewing and providing feedback on physics, chemistry, and biology experiment reports based on a multimodal large-scale model. This invention achieves high-precision, interpretable, and evolvable intelligent review of physics, chemistry, and biology experiment reports by deeply integrating a multimodal large-scale model, structured knowledge representation, retrieval enhancement generation, and an educational intelligent feedback mechanism. By fine-tuning a visual encoder with subject symbol perception capabilities and a multimodal large-scale language model, the system can accurately identify key components, connections, and drawing specifications in hand-drawn images, and perform logical reasoning based on structured scoring rules, effectively solving the problem that traditional models cannot handle drawing questions or have poor robustness to non-standard answers. The scoring criteria are decomposed into independent, composable knowledge units and a multimodal vector knowledge base is constructed, supporting semantic retrieval and matching of scoring criteria. It can also be added, deleted, and modified online without retraining the large-scale model, greatly reducing maintenance costs. The system not only outputs scores but also automatically identifies error types and generates targeted teaching suggestions based on the subject knowledge base, upgrading review from a scoring tool to a teaching assistant, effectively supporting formative assessment and personalized learning intervention. The system automatically filters low-confidence samples through a joint confidence mechanism and submits them to teachers for review. It also uses multimodal embedding and clustering algorithms to automatically discover new error patterns from manually corrected data. After review, these patterns are injected into the knowledge base, enabling continuous expansion of error cognition and self-iteration of system capabilities. This allows the review system to adapt to changes in teaching practice over a long period of time. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating a method for reviewing and providing feedback on physics, chemistry, and biology experiment reports based on a multimodal large model, according to an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of the structure of a physics, chemistry, and biology experiment report review and feedback system based on a multimodal large model according to an embodiment of the present invention. Detailed Implementation

[0060] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0061] The purpose of this invention is to overcome the shortcomings of existing technologies, such as low efficiency, poor accuracy, lack of teaching feedback, and difficulty in system evolution, by proposing a method and system for reviewing and providing feedback on physics, chemistry, and biology experiment reports based on a multimodal large model. Specifically, in the embodiments of this invention:

[0062] 1. Construct a multimodal large-scale model-specific fine-tuning and architecture optimization mechanism for experimental report review scenarios. The design is based on a visual encoder-text decoder architecture, combined with a subject-specific symbol recognition adapter module and a lightweight cross-attention mechanism, to perform dual fine-tuning of the visual encoder and the embedded model. This retains general visual capabilities while enhancing the fine-grained understanding of key components, connections, and drawing specifications in physics, chemistry, and biology experimental diagrams.

[0063] 2. Construct a structured and scalable multimodal scoring knowledge base and dynamic retrieval mechanism. It is the first to decompose scoring criteria into independent knowledge units and construct structured knowledge documents containing text and image paths; it generates joint vectors through a domain-adaptive multimodal embedding model, stores them in a vector database, supports online updates without retraining large models, and achieves efficient and accurate semantic retrieval and scoring criterion matching.

[0064] 3. Design a multi-stage integrated analysis process for intelligent evaluation of drawing questions. For drawing questions that traditional AI struggles to handle, a two-stage scoring mechanism is designed, combining lightweight visual analysis and multimodal large-scale model reasoning. First, the fine-tuned visual model extracts component existence and spatial information. Then, structured metadata and labeled images are input into the fine-tuned large-scale model for rule compliance judgment, effectively bridging the semantic gap between pixel-level images and abstract scoring rules.

[0065] 4. Construct a closed-loop teaching feedback and error pattern self-evolution system. This system not only outputs scores but also generates structured teaching feedback through multi-agent module collaboration. Furthermore, it introduces a joint confidence assessment mechanism to trigger manual review and automatically clusters low-confidence samples to discover new error patterns. After teacher review, these patterns are injected into the knowledge base, forming a continuous evolutionary closed loop of review-feedback-learning-optimization.

[0066] refer to Figure 1 ,like Figure 1 The figure shown is a method for reviewing and providing feedback on physics, chemistry, and biology experiment reports based on a multimodal large model, provided by an embodiment of the present invention. The method includes the following steps:

[0067] S100: Obtain experimental report question data, candidate answer data, and scoring standard data. The candidate answer data includes text answer data or image answer data.

[0068] In this embodiment, the system obtains the answer to a question submitted by a candidate through an information-based examination platform, and acquires question-answer-scoring criterion data, where the answer can be text or a standard image. The question data includes the question text and question type information, and the scoring criterion data includes the score allocation and scoring details.

[0069] S200, the experimental report question data is input into a multimodal embedding model for vectorization processing to generate question embedding vectors. Based on the question embedding vectors, semantically similar knowledge records are retrieved from a preset scoring knowledge base as scoring basis data. The scoring knowledge base stores joint embedding vectors of structured scoring knowledge documents.

[0070] In this embodiment, the text of the question submitted by the examinee is vectorized, and the most semantically similar knowledge records are retrieved from the scoring knowledge base as the scoring basis for the question. Similarity calculation uses cosine similarity, BM25, or TF1df methods. The scoring knowledge base supports online addition, deletion, and modification operations, and updates take effect without requiring retraining of the large model.

[0071] S300, when the candidate's answer data is text answer data, the experimental report question data, the text answer data, the scoring basis data, and the scoring module system prompts are integrated into a text scoring instruction. The text scoring instruction is input into the fine-tuned multimodal large language model for scoring reasoning, and a preliminary review result is output. The preliminary review result includes a score field and a scoring basis field.

[0072] In this embodiment, the intelligent agent process includes a scoring module, an error identification and classification module, and a teaching suggestion generation module. Different modules have corresponding system prompts. For text-based questions, the question-answer-scoring standard metadata and the scoring module's system prompts are integrated into a dedicated instruction template and sent to the large model for scoring.

[0073] S400, when the candidate's answer data is image answer data, the image answer data is input into a lightweight visual model for component detection, and component existence information and component annotation map are output. Metadata is generated based on the component existence information and the scoring basis data. The component annotation map, the metadata, and the scoring module system prompts are integrated into an image scoring instruction. The image scoring instruction is input into the fine-tuned multimodal large language model for scoring reasoning, and a preliminary evaluation result is output. The component existence information includes center coordinates, bounding box, and component type.

[0074] In this embodiment, for drawing questions, a lightweight visual model is first used to perform visual analysis on the candidate's image, generating component annotation maps and outputting the existence information of each component, including center coordinates, bounding box, and component type. Then, the component existence information from the visual analysis results is combined with the question-answer-scoring criteria to generate metadata. Finally, the component annotation maps, metadata, and system prompts from the scoring module are integrated into a dedicated instruction template and input into the finely tuned large-scale visual model for scoring.

[0075] S500, based on the scoring criteria field and the candidate's answer data, calls the error identification module to extract error semantic fragments, combines the error pattern knowledge base to classify and match the error semantic fragments, and outputs the error type field.

[0076] In this embodiment, the error identification module loads a dedicated system prompt word template to guide the large model in extracting error semantic fragments from the scoring criteria. Combining this with an error pattern knowledge base (including static rules and self-discovered patterns), errors are categorized and matched, outputting structured error types with multiple labels, such as missing components or connection errors.

[0077] S600, based on the score field, the scoring basis field and the error type field, the teaching suggestion generation module is invoked to generate the teaching suggestion field.

[0078] In this embodiment, the teaching suggestion generation module loads subject-adapted suggestion generation prompt templates, forcibly references knowledge point tags and correction strategies, guides the large model to generate personalized teaching feedback, and outputs structured field teaching suggestions, such as "Please note that the circuit diagram must be a closed loop".

[0079] S700, summarize the score field, the scoring basis field, the error type field, and the teaching suggestion field to form a complete review result.

[0080] S800: Calculate the joint confidence level based on the complete review result. When the joint confidence level is less than a preset threshold, mark the complete review result as pending manual review and push it to the teacher's end; otherwise, output the complete review result directly.

[0081] In this embodiment, the joint confidence score for generating the review is defined as a weighted combination of the historical manual confirmation rate of the knowledge unit, the difficulty factor of the question, and the consistency score of the student's historical performance. The preset threshold is dynamically set according to the subject. If the joint confidence score is less than the threshold, the question is marked as requiring manual review, and the complete review package is pushed to the teacher's end for scoring correction and recorded in the manual review record; otherwise, the final review result is returned directly.

[0082] Specifically, the joint confidence formula is as follows: ;in, For joint confidence level; The historical manual confirmation rate of the knowledge units used (statistics of the most recent 100 reviews are obtained through a sliding window); This is the difficulty factor for the question, calculated as the historical average score rate for that question (default is 0.7 if no data is available). The consistency score for a student's historical performance is calculated based on the standard deviation of the student's scores on the same type of question in the last three tests.

[0083] In some embodiments, before inputting the experimental report question data into a multimodal embedding model for vectorization processing, the method further includes the step of constructing the scoring knowledge base:

[0084] For short answer and fill-in-the-blank questions and drawing questions in physics, chemistry and biology lab reports, a structured scoring standard is developed. The structured scoring standard includes question scores, reference answers and scoring details. The scoring standard for drawing questions is further subdivided into independent knowledge units, which include rules on the existence of components, rules on connection relationships, rules on drawing standardization, and rules on the accuracy of annotation.

[0085] Construct a structured scoring knowledge document. Each of the structured scoring knowledge documents includes a question text field, a reference answer field, a scoring criterion field, and a question score field. The reference answer field stores text content or image paths.

[0086] The structured scoring knowledge document is cleaned and preprocessed, the text content is segmented and stop word removed, and the image content corresponding to the image path is uniformly cropped to a preset pixel size.

[0087] The preprocessed structured scoring knowledge document is jointly vectorized using a domain-adaptive multimodal embedding model to generate a joint embedding vector. The joint embedding vector is then stored in a vector database to form a dynamically updatable scoring knowledge base.

[0088] In this embodiment, the reference answer for the drawing question must include a standard image, a textual description of the image, and subject-specific explanations of key component symbols, such as the polarity marking rules for batteries in circuit diagrams or the connection specifications for conduits in chemical devices. Image content is uniformly cropped to 224×224 pixels, and the original image is replaced with the server storage path. The vector database can be FAISS, Milvus, or PGVector.

[0089] In some embodiments, the step of jointly vectorizing the preprocessed structured scoring knowledge document using a domain-adaptive multimodal embedding model to generate a joint embedding vector includes:

[0090] Model Construction: A dedicated embedding model is designed based on a multimodal large language model architecture, which includes a visual encoder, a text encoder, and a cross-modal alignment module; the cross-modal alignment module adopts a learnable projection layer and a lightweight cross-attention layer;

[0091] Domain fine-tuning: End-to-end fine-tuning was performed on a self-constructed physics, chemistry, and biology experiment report image-text pair dataset added to the public image-text pair dataset; the fine-tuning strategy was to fix the backbone parameters of the visual encoder and only fine-tune the projection layer and the cross-attention layer; the optimization objective function was a weighted combination of contrastive learning loss and triplet boundary loss; domain enhancement was achieved by setting weighted loss weights for physics, chemistry, and biology experiment element-term pairs to strengthen key semantic alignment.

[0092] Embedding generation: The model is fine-tuned by inputting the text fields of the knowledge record and the corresponding image, and the output is a fixed-dimensional L2 normalized joint embedding vector.

[0093] In some embodiments, before inputting the image answer data into the lightweight visual model for component detection, the method further includes a step of fine-tuning the lightweight visual model:

[0094] We selected a self-supervised pre-trained visual Transformer network as the basic model, constructed a training dataset of symbols for physics, chemistry and biology, and used a generative adversarial network to convert standard images into various hand-drawn styles to improve data diversity.

[0095] A parallel adapter module is added to the basic model, and the adapter module adopts a fully convolutional structure.

[0096] A gating unit is added before the output features of the adapter module are fused with the output features of the base model;

[0097] The context features of the base model and the detail features of the adapter module are subjected to bilinear pooling.

[0098] Freeze the backbone parameters of the base model, train only the parameters of the adapter module, and use a weighted combination of contrastive loss, bounding box regression loss and classification cross-entropy loss as the loss function to complete the fine-tuning of the lightweight vision model.

[0099] In this embodiment, each module of the adapter module outputs features at the same scale as the attention module of the base model, focusing on the details of the symbolic pattern. The gating unit is activated only when processing relevant symbolic data features. This results in a fine-tuned detector for the scientific symbol domain, while also possessing general visual feature extraction capabilities.

[0100] In some embodiments, calculating the joint confidence level based on the complete review results includes:

[0101] Obtain the historical manual confirmation rate of the knowledge unit corresponding to the scoring basis data, and use a sliding window to count the proportion of manual confirmations in the last preset number of reviews;

[0102] Obtain the historical average score rate of the questions corresponding to the experimental report questions; if no data is available, use the default value.

[0103] Obtain the consistency score of the student's historical performance corresponding to the candidate's answer data, and calculate it based on the standard deviation of the student's scores on similar questions in the most recent preset number of times;

[0104] The joint confidence score is generated by weighting and summing the historical manual confirmation rate, the historical average score rate, and the historical performance consistency score.

[0105] In some embodiments, the method further includes a step of error mode self-evolution:

[0106] For all low-confidence or manually reviewed samples, extract the candidate's answer embedding vector;

[0107] Obtain all error samples for each question type, extract the candidate's answer-error type text or image data, and generate embedding vectors using a multimodal embedding model;

[0108] A density clustering algorithm with cosine distance as the distance metric is used to cluster and merge erroneous clusters and discover new erroneous clusters;

[0109] Generate candidate error patterns, including summaries, typical examples, and causal inferences, and submit them to teachers for review;

[0110] Once approved, the error pattern knowledge base will be automatically injected and associated with the corresponding knowledge point.

[0111] In this embodiment, a joint confidence mechanism is used to automatically screen low-confidence samples for review by teachers. Multimodal embedding and clustering algorithms are used to automatically discover new error patterns from manually corrected data. After review, these patterns are injected into the knowledge base, enabling continuous expansion of error cognition and self-iteration of system capabilities.

[0112] In some embodiments, before inputting the text scoring instruction or the image scoring instruction into the fine-tuned multimodal large language model for scoring inference, the method further includes a step of fine-tuning the multimodal large language model:

[0113] We construct a supervised fine-tuning dataset for experimental image and text scoring instructions, and reuse the feature extraction backbone of the fine-tuned lightweight visual model as the visual word segmenter of the multimodal large model. The word segmenter outputs a visual token sequence, which is concatenated with the text token after passing through a projection layer and input into the text decoder to obtain the output.

[0114] This involves fine-tuning only the parameters of the projection layer while freezing the remaining parameters, thus completing the supervised fine-tuning of the large visual model.

[0115] The following is a specific embodiment provided by the present invention:

[0116] Step 1: Data preparation;

[0117] We collected a dataset of middle school physics, chemistry, and biology experiment reports, including image data such as physics circuit diagrams, chemical experimental apparatus diagrams, and biological structure diagrams, as well as text data such as descriptions of experimental phenomena and principle analyses. The data was divided into training, validation, and test sets.

[0118] Step 2: Knowledge base construction;

[0119] A structured scoring standard was developed, subdividing the scoring criteria for drawing questions into independent knowledge units such as component existence rules, connection relationship rules, drawing standardization rules, and annotation accuracy rules. A structured scoring knowledge document was constructed, including fields such as question text, reference answer, scoring standard, and question score. The knowledge document underwent data cleaning, including text segmentation and stop word removal, and images were uniformly cropped to 224×224 pixels.

[0120] Step 3: Model fine-tuning;

[0121] Visual model fine-tuning: A self-supervised pre-trained visual Transformer was selected as the base model, a parallel adapter module was added, the backbone parameters were frozen, and only the adapter module was trained. The training was performed using a weighted combination of contrast loss, bounding box regression loss, and classification cross-entropy loss.

[0122] Multimodal large model fine-tuning: Construct an experimental image-text scoring instruction supervised fine-tuning dataset, use the fine-tuned visual model as a visual word segmenter, freeze the parameters of the visual word segmenter and text decoder, and only fine-tune the projection layer parameters.

[0123] Embedded model fine-tuning: Based on a multimodal large language model architecture, the backbone parameters of the visual encoder are fixed, and only the projection layer and cross attention layer are fine-tuned. A weighted combination of contrastive learning loss and triplet boundary loss is adopted, and weighted loss weights are set for the physical, chemical and biological experimental element-term pairs.

[0124] Step 4: System Deployment;

[0125] Deploy the fine-tuned model to the server and build a scoring knowledge base vector database. Configure system prompt words for the intelligent agent modules, including dedicated prompt word templates for the scoring module, error recognition module, and teaching suggestion generation module.

[0126] Step 5: Review Process;

[0127] Candidates submit their answers through an information-based examination platform. The system retrieves the question-answer-scoring criteria data. After vectorizing the question text, it searches for similar knowledge records in the scoring knowledge base as the basis for scoring. Depending on the answer type, it selects either a text-based scoring process or an image-based scoring process, calls a finely tuned multimodal large language model for scoring reasoning, and outputs preliminary review results. It then sequentially calls the error identification module and the teaching suggestion generation module to summarize and form a complete review result. Finally, it calculates the joint confidence score to determine whether to send the result for manual review.

[0128] Step 6: Error Mode Evolution;

[0129] Low-confidence samples and manually reviewed samples are periodically extracted, vectorized, and clustered to identify new error patterns. After teacher review, these patterns are injected into the error pattern knowledge base to enable the system to self-evolve.

[0130] Compared with related technologies, the present invention has the following advantages:

[0131] 1. Significantly improves the accuracy and fine-grained understanding of lab report reviews:

[0132] By fine-tuning a visual encoder with subject-specific symbol perception capabilities and a multimodal large language model, the system can accurately identify key elements, connections, and drawing specifications in hand-drawn images. Combined with structured scoring rules, it performs logical reasoning, effectively solving the problem that traditional models cannot handle drawing questions or have poor robustness to non-standard answers, thus significantly improving scoring accuracy, especially in complex experimental illustration scenarios.

[0133] 2. Achieve structured, modular, and dynamically maintainable scoring criteria:

[0134] The scoring criteria are broken down into independent, composable knowledge units and a multimodal vector knowledge base is constructed, supporting semantic retrieval and matching of scoring criteria. This knowledge base can be added, deleted, and modified online without retraining a large model, greatly reducing maintenance costs. At the same time, it ensures that the scoring logic is transparent and traceable, meeting the requirements of fairness and consistency in educational assessment.

[0135] 3. Generate personalized, instruction-oriented structured feedback to promote a closed learning loop for students:

[0136] The system not only outputs scores but also automatically identifies error types, such as reversed battery polarity or a conduit not being submerged in liquid. Based on a subject-specific knowledge base, it generates targeted teaching suggestions, such as the prerequisite for current flow in a closed circuit. This integrated diagnostic-attribution-guidance feedback mechanism upgrades assessment from a scoring tool to a teaching assistant, effectively supporting formative evaluation and personalized learning intervention.

[0137] 4. Construct a self-evolving evaluation system to continuously improve the system's intelligence level:

[0138] The system automatically filters low-confidence samples through a joint confidence mechanism, which are then submitted to teachers for review. Furthermore, multimodal embedding and clustering algorithms are used to automatically identify new error patterns from manually corrected data. These patterns are then added to the knowledge base after review, enabling continuous expansion of error recognition and self-iteration of system capabilities. This mechanism allows the review system to adapt to changes in teaching practice over the long term.

[0139] refer to Figure 2 This invention also provides a physics, chemistry, and biology experiment report review and feedback system based on a multimodal large model, comprising:

[0140] At least one processor;

[0141] At least one memory for storing at least one program;

[0142] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0143] The content of the above method embodiments is applicable to this embodiment. The specific functions implemented in this embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. Therefore, they will not be repeated here.

[0144] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the above-described method.

[0145] It is worth noting that, since the computer-readable storage medium of the present invention is capable of executing the methods of any of the above embodiments, the specific implementation methods and technical effects of the computer-readable storage medium of the present invention can be referred to the specific implementation methods and technical effects of the methods of any of the above embodiments.

[0146] Furthermore, embodiments of the present invention also disclose a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the methods described above.

[0147] Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0148] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0149] Although the description of this disclosure has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment, but should be considered as effectively covering the intended scope of this disclosure by referring to the appended claims and taking into account the broad possible interpretations of these claims provided by the prior art. Furthermore, the foregoing description of this disclosure with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this disclosure that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A method for reviewing and providing feedback on physics, chemistry, and biology experiment reports based on a multimodal large model, characterized in that, The method includes the following steps: Acquire experimental report question data, student answer data, and scoring standard data, wherein the student answer data includes text answer data or image answer data; The experimental report question data is input into a multimodal embedding model for vectorization processing to generate question embedding vectors. Based on the question embedding vectors, semantically similar knowledge records are retrieved from a preset scoring knowledge base as scoring basis data. The scoring knowledge base stores joint embedding vectors of structured scoring knowledge documents. When the candidate's answer data is text answer data, the experimental report question data, the text answer data, the scoring basis data, and the scoring module system prompts are integrated into a text scoring instruction. The text scoring instruction is then input into a fine-tuned multimodal large language model for scoring reasoning, and a preliminary review result is output. The preliminary review result includes a score field and a scoring basis field. When the candidate's answer data is image-based, the image-based answer data is input into a lightweight visual model for component detection, outputting component presence information and component annotation maps. Metadata is generated based on the component presence information and the scoring criteria data. The component annotation maps, the metadata, and the system prompts from the scoring module are integrated into an image scoring instruction. The image scoring instruction is input into the finely tuned multimodal large language model for scoring reasoning, and a preliminary evaluation result is output. The component presence information includes center coordinates, bounding boxes, and component types. Based on the scoring criteria field and the candidate's answer data, the error identification module is called to extract error semantic fragments, and the error semantic fragments are classified and matched in combination with the error pattern knowledge base to output the error type field; Based on the score field, the scoring basis field, and the error type field, the teaching suggestion generation module is invoked to generate a teaching suggestion field; The score field, the scoring basis field, the error type field, and the teaching suggestion field are summarized to form a complete review result; Calculate the joint confidence level based on the complete review results. When the joint confidence level is less than a preset threshold, mark the complete review results as pending manual review and push them to the teacher's end; otherwise, output the complete review results directly.

2. The method according to claim 1, characterized in that, Before inputting the experimental report title data into the multimodal embedding model for vectorization processing, the process also includes: For short answer and fill-in-the-blank questions and drawing questions in physics, chemistry and biology lab reports, a structured scoring standard is developed. The structured scoring standard includes question scores, reference answers and scoring details. The scoring standard for drawing questions is further subdivided into independent knowledge units, which include rules on the existence of components, rules on connection relationships, rules on drawing standardization, and rules on the accuracy of annotation. Construct a structured scoring knowledge document. Each of the structured scoring knowledge documents includes a question text field, a reference answer field, a scoring criterion field, and a question score field. The reference answer field stores text content or image paths. The structured scoring knowledge document is cleaned and preprocessed, the text content is segmented and stop word removed, and the image content corresponding to the image path is uniformly cropped to a preset pixel size. The preprocessed structured scoring knowledge document is jointly vectorized using a domain-adaptive multimodal embedding model to generate a joint embedding vector. The joint embedding vector is then stored in a vector database to form a dynamically updatable scoring knowledge base.

3. The method according to claim 2, characterized in that, The method of employing a domain-adaptive multimodal embedding model to jointly vectorize the preprocessed structured scoring knowledge document and generate a joint embedding vector includes: The title text field is input into the text encoder to generate a text embedding vector. The image content corresponding to the image path is input into the visual encoder to generate a visual embedding vector. The visual encoder adopts a visual Transformer architecture based on adapter fine-tuning. The visual embedding vector and the text embedding vector are mapped to the same feature space through a cross-modal alignment module, which employs a learnable projection layer and a lightweight cross-attention layer. The mapped visual embedding vector and the text embedding vector are fused to output a joint embedding vector, which is a fixed-dimensional vector that has been L2 normalized.

4. The method according to claim 1, characterized in that, Before inputting the image answer data into the lightweight visual model for component detection, the method further includes: A self-supervised pre-trained visual Transformer network was selected as the basic model to construct a training dataset of symbols for physics, chemistry and biology disciplines. The training dataset includes physical circuit diagrams, chemical device diagrams and biological structure diagrams. A parallel adapter module is added to the base model. The adapter module adopts a fully convolutional structure. The output features of the adapter module are at the same scale as the output features of the attention module of the base model. The output features of the adapter module focus on the detailed information of the symbol pattern. A gating unit is added before the output features of the adapter module are fused with the output features of the base model. The gating unit is used to control the activation state of the adapter module. The context features of the base model and the detail features of the adapter module are subjected to bilinear pooling to enhance the correlation between features; Freeze the backbone parameters of the base model, train only the parameters of the adapter module, and use a weighted combination of contrastive loss, bounding box regression loss and classification cross-entropy loss as the loss function to fine-tune the base model and obtain a lightweight visual model.

5. The method according to claim 1, characterized in that, The calculation of joint confidence based on the complete review results includes: Obtain the historical manual confirmation rate of the knowledge unit corresponding to the scoring basis data, the historical average score rate of the questions corresponding to the experimental report question data, and the historical performance consistency score of the students corresponding to the candidate answer data; The historical manual confirmation rate, the historical average score rate, and the historical performance consistency score are weighted and summed to generate a joint confidence score. The weight coefficients of the weighted summation are dynamically set according to the subject type.

6. The method according to claim 1, characterized in that, The method further includes: Extract the candidate answer data marked as awaiting manual review, and the review result data after manual review and correction, as low-confidence sample data; The low-confidence sample data is vectorized to generate sample embedding vectors; A density clustering algorithm based on cosine distance is used to perform cluster analysis on the sample embedding vectors to identify approximate error clusters and new error clusters; Candidate error patterns are generated based on clustering results. Each candidate error pattern includes an error summary, typical examples, and cause speculation. The candidate error patterns are then pushed to the teacher's end for review. When the candidate error pattern passes the review, the candidate error pattern is automatically injected into the error pattern knowledge base and associated with the corresponding knowledge point.

7. The method according to claim 4, characterized in that, Before inputting the text scoring instruction or the image scoring instruction into the fine-tuned multimodal large language model for scoring inference, the method further includes: Construct a supervised fine-tuning dataset for experimental graphic scoring instructions. Each sample in the dataset contains the question text, the examinee's answer, the scoring criteria, and the standard review result. The lightweight visual model is used as a visual tokenizer, and the visual tokenizer outputs a sequence of visual tokens. The visual token sequence is processed by a projection layer and then concatenated with a text token. The result is then input into a text decoder. The projection layer employs a projection mechanism similar to a contrastive language image pre-training architecture. The parameters of the visual word segmenter and the text decoder are frozen, and only the parameters of the projection layer are fine-tuned to complete the supervised fine-tuning of the multimodal large language model, resulting in the fine-tuned multimodal large language model.

8. The method according to claim 1, characterized in that, Based on the scoring criteria field and the candidate's answer data, the error identification module is invoked to extract error semantic fragments. These fragments are then categorized and matched using an error pattern knowledge base, and an error type field is output, including: The error identification module loads a dedicated system prompt word template, which guides the multimodal large language model to extract erroneous semantic fragments from the scoring criteria field that do not match the candidate's answer data. The extracted error semantic fragments are matched with static rules and historical self-discovery patterns in the error pattern knowledge base to determine the error type label; The output contains structured data including the error type field, which supports multi-label format.

9. A physics, chemistry, and biology experiment report review and feedback system based on a multimodal large model, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images