Multi-modal classroom progress evaluation and teaching assistance method based on end-cloud collaborative architecture

Abstract

The invention discloses a multi-modal classroom progress evaluation and teaching assistance method based on an end-cloud collaborative architecture. Synchronously acquiring PPT pictures and facial expression data sets of teachers and text voices of the teachers at the classroom end side; splicing all the collected data and a preset prompt text to obtain a final input text; inputting the final input text into a pre-trained InternVL3 model for reasoning to obtain a reasoning result, pushing the reasoning result to a teacher terminal, and performing teaching guidance by a teacher according to the reasoning result; and repeating the above steps after a preset time period, so that the teacher continues teaching guidance according to the new reasoning result. The teaching rhythm is judged by fusing the teaching content of the teacher, the behavior response of the student group and the voice information, accurate and operable teaching feedback is provided for the teacher, and exercises meeting the current knowledge point and difficulty requirements are recommended especially when the progress is abnormal, so that the teacher is helped to dynamically adjust the teaching rhythm, and the classroom teaching efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of smart education and artificial intelligence, and specifically relates to a multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture. Background Technology

[0002] In recent years, multimodal large models (such as InternVL and LLaVA) have demonstrated great potential in smart education scenarios due to their outstanding capabilities in visual-language joint understanding. These models can process image, text, and speech information simultaneously and are being tested for tasks such as classroom behavior analysis, attention detection, and understanding of teaching content. Meanwhile, improvements in edge computing power have made it possible to perform lightweight perception locally in the classroom, laying the foundation for low-latency, high-privacy educational AI applications.

[0003] However, there are still two major challenges when applying multimodal large models directly to classroom teaching assistance.

[0004] First, existing systems primarily focus on recognizing individual student states (such as facial expression classification and attention scoring), lacking a macro-level assessment of the overall teaching pace and knowledge transfer efficiency. What teachers truly need is not "whether a particular student is distracted," but rather "whether the current pace of explanation matches the class's overall comprehension level." Existing solutions struggle to extract effective judgments about teaching progress from multi-source, heterogeneous data including PPT content, student responses, and the speaker's voice, resulting in feedback that is detached from actual teaching needs.

[0005] Secondly, even if some systems can output conclusions such as "progress is too fast," subsequent interventions still heavily rely on large models directly generating teaching content (such as questions and prompts), posing risks of unreliable, uncontrollable, and unauditable content. Educational scenarios demand extremely high accuracy and security; questions freely generated by large models may contain knowledge errors, difficulty mismatches, or ambiguous expressions, severely impacting teaching effectiveness. Furthermore, intervention strategies are often fixed rules, unable to be dynamically adjusted based on the type of knowledge point, student performance (such as confusion, bewilderment, or silence), lacking flexibility and adaptability. Summary of the Invention

[0006] To address the problems existing in the background technology, this invention provides a multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture, solving two major technical problems in the prior art: inaccurate perception of teaching progress and unreliable intervention content. The purpose of this invention is to achieve real-time, interpretable judgment of the overall teaching pace by integrating teacher's teaching content, student group behavioral responses, and speech and semantic information. Based on this, and combining a combination matching pool of overall student status tags and teaching progress tags with a question bank retrieval mechanism, it provides teachers with safe, accurate, and operable teaching feedback. Especially when progress is abnormal, it recommends exercises that match the current knowledge points and difficulty levels, thereby helping teachers dynamically adjust the teaching pace and improve classroom teaching efficiency.

[0007] The technical solution adopted in this invention is: I. A Multimodal Classroom Progress Assessment and Teaching Assistance Method Based on End-to-Cloud Collaborative Architecture: S1. Simultaneously collect the teacher's PPT slides, facial expression dataset, and text / voice data on the classroom end.

[0008] S2. Combine all the data collected in step S1 with the preset prompt text to obtain the final input text.

[0009] S3. Input the final input text into the pre-trained InternVL3 model for inference to obtain the final inference result. Push the final inference result to the teacher's terminal, and the teacher provides teaching guidance based on the inference result.

[0010] S4. After a preset time period, repeat steps S1-S3, so that the teacher can continue to provide teaching guidance based on the new reasoning results.

[0011] Step S1 specifically involves: S11. Obtain the current frame image, including the faces of all students, during the class, and use the current frame image as the group expression analysis frame.

[0012] S12. Perform face detection on the group expression analysis frames to obtain the detection box image of each student's face expression. The detection box images of all students' face expressions are summarized to obtain the face expression dataset.

[0013] S13. Obtain the PPT screen currently being presented by the teacher during class.

[0014] S14. Obtain the teacher's voice from the previous several seconds to the current moment, and convert the teacher's voice into text-to-speech.

[0015] Step S2 specifically involves: S21. Concatenate all images, PPT slides, and text / speech data in the facial expression dataset to obtain the text information.

[0016] S22. Combine the preset prompt text with the text information to obtain the final input text.

[0017] The preset prompt text is obtained by sequentially analyzing the detection box image of each student's facial expression, PPT screen and text voice in the input text information, and then outputting a combination of the current status label of the overall students and the teaching progress label. The combination of the current overall student status label and teaching progress label includes: [Status label: Focused, Teaching progress label: Normal], [Status label: Confused, Teaching progress label: Too fast], [Status label: Relaxed, Teaching progress label: Too slow], [Status label: Silent, Teaching progress label: Too fast] and [Status label: Silent, Teaching progress label: Too slow].

[0018] Step S3 specifically involves: S31. Input the final input text into the pre-trained InternVL3 model for inference to obtain a combination of the current overall student status label and teaching progress label.

[0019] S32. The pre-trained InternVL3 model outputs the final inference result based on the combination of the current overall student status label and the teaching progress label.

[0020] S33. The reasoning results are pushed to the classroom terminal, and the teacher provides teaching guidance based on the obtained reasoning results.

[0021] Step S32 specifically involves: When the obtained combination is [state label: focused, teaching progress label: normal], the pre-trained InternVL3 model outputs the final inference result: the teaching progress is appropriate, the teacher can continue teaching and no adjustments are needed.

[0022] When the obtained combination is [Status label: Confused, Teaching progress label: Too fast], the pre-trained InternVL3 model outputs the final inference result: suggesting that the teacher should elaborate on the knowledge points being explained.

[0023] When the obtained combination is [Status label: Easy, Teaching progress label: Too slow], the pre-trained InternVL3 model outputs the final inference result: The students have mastered the current content, and it is recommended that the teacher speed up the pace of the explanation.

[0024] When the obtained combination is [Status label: Silent, Teaching progress label: Too fast] or [Status label: Silent, Teaching progress label: Too slow], the pre-trained InternVL3 model first outputs the structured fields of the proposed recommended questions. Then, based on the structured fields of the proposed recommended questions, the pre-trained InternVL3 model calls the recommended question acquisition module to obtain the question stem, options, and answers of the recommended questions. The pre-trained InternVL3 model outputs the obtained question stem, options, answers, and attached teaching suggestions as the final reasoning result.

[0025] The structured fields include the subject category of the recommended questions, the name of the knowledge point, the difficulty value of the recommended questions, the question type, and the natural language description of the questions.

[0026] The recommended question acquisition module is implemented according to the following steps: D1. The pre-trained Sentence-BERT model is used to perform semantic encoding on the structured fields and knowledge point names respectively, so as to obtain the semantic vectors of the structured fields and the semantic vectors of the knowledge points respectively.

[0027] D2. Expand the recommended question difficulty values ​​into a one-dimensional vector.

[0028] D3. Concatenate the semantic vector, the knowledge point semantic vector, and the recommended question difficulty value of the one-dimensional vector in sequence to generate the final question requirement vector.

[0029] D4. Based on the preset structured question bank and question demand vector, obtain recommended questions, and obtain the question stem, options and answers of the recommended questions.

[0030] The specific steps for obtaining recommended questions based on a pre-set structured question bank and question requirement vector are as follows: In the pre-set structured question bank, each question has a pre-constructed corresponding embedding vector, the construction method of which is consistent with the question requirement vector. The recommended questions are obtained by processing according to the following formula: in, Represents the vector required by the problem; For indexing; This indicates the first question in the pre-defined structured question bank. Embedding vectors corresponding to each question; In order to make Embedded vector reaches maximum The corresponding questions also indicate recommended questions; This indicates taking the maximum value; Indicates cosine similarity; This represents the total number of questions in the structured question bank. Table L2 norm; Represents the vector of requirements for the problem. The transpose of .

[0031] II. A Multimodal Classroom Progress Assessment and Teaching Assistance System Based on End-to-Cloud Collaborative Architecture The classroom-side data acquisition module synchronously collects the teacher's PPT slides, facial expression dataset, and text / speech data on the classroom end. The input text acquisition module concatenates all data collected by the classroom-side data acquisition module with preset prompt text to obtain the final input text. The model inference and teacher-side result acquisition module inputs the final input text into a pre-trained InternVL3 model for inference, obtains the inference result, and pushes the inference result to the teacher's terminal, allowing the teacher to provide teaching guidance based on the inference result.

[0032] The beneficial effects of this invention are: 1. The method of this invention aims to solve the problems of existing smart education systems being unable to perceive the overall teaching pace in real time and relying on unreliable large-scale model generation for intervention content. By constructing a closed-loop framework of "lightweight edge perception—cloud-based multimodal reasoning—combined strategy execution," this invention can accurately determine whether the current teaching progress is too fast, too slow, or too moderate, and provide teachers with explainable, operable, and traceable teaching suggestions in abnormal situations. In particular, it can recommend precise exercises that meet the requirements of knowledge points, difficulty, and question types, effectively improving the pertinence and timeliness of classroom teaching.

[0033] 2. Extensive simulation and real classroom tests show that the present invention has significant effects in helping teachers dynamically adjust the pace of instruction, reduce students' cognitive load, and improve classroom participation, while also demonstrating good privacy protection capabilities and system robustness.

[0034] 3. This invention significantly improves the accuracy of judging teaching progress by introducing the InternVL3 multimodal reasoning mechanism driven by the chain-of-thought of prompt text, combined with PPT content, group facial expressions (such as confusion, bewilderment, silence, etc.) and speech semantics.

[0035] 4. Furthermore, this invention decouples the intervention action into a two-stage process: "combined matching of overall student status labels and teaching progress labels + question bank vector retrieval," avoiding the risks of knowledge errors and difficulty mismatches caused by directly generating questions from a large model. Question bank retrieval is based on SBERT fusion embedding and structured metadata, ensuring that the returned questions highly match teaching needs in terms of semantics, knowledge scope, and cognitive difficulty. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention.

[0037] Figure 2 This is a block diagram of the system of the present invention. Detailed Implementation

[0038] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

[0039] Example 1: like Figure 1 As shown, the multimodal classroom progress assessment and teaching support method of this embodiment includes the following steps: S1. Simultaneously collect the teacher's PPT slides, facial expression dataset, and text / voice data on the classroom end.

[0040] This step implements part of the lightweight sensing framework on the edge side.

[0041] S11. Acquire the current frame image, including the faces of all students, through the cameras deployed in the classroom during class, and use the current frame image as the group expression analysis frame.

[0042] S12. Perform face detection on the group expression analysis frames to obtain the detection box image of each student's face expression. The detection box images of all students' face expressions are summarized to obtain the face expression dataset.

[0043] In practice, the process of acquiring facial expression datasets is legal and compliant.

[0044] S13. Capture the PPT slides currently being presented by the teacher during class by taking a screenshot.

[0045] S14. Obtain the teacher's speech from several consecutive seconds before the current time to the current time, and convert the teacher's speech into text speech using a locally deployed pre-trained Whisper model.

[0046] S2. In the cloud, all the data collected in step S1 and the preset prompt text are combined to obtain the final input text.

[0047] S21. Concatenate all images, PPT slides, and text / speech data in the facial expression dataset to obtain the text information Context.

[0048] In practice, the text information Context = the image of the first student's facial expression detection box in the facial expression dataset:[ The second student's facial expression detection bounding box image in the facial expression dataset: [ ];...;The bounding box image of the nth student's facial expression in the facial expression dataset:[ ]; PPT slides: [ Teacher's text / voice: [ The semicolon (;) indicates a splicing operation.

[0049] S22. Concatenate the preset prompt text Prompt with the text information Context to obtain the final input text.

[0050] The preset prompt text Prompt is a combination of the detection box image of each student's facial expression, PPT screen and text voice in the input text information, and then outputs the current status label of the overall students and the teaching progress label.

[0051] This invention significantly improves the accuracy of judging teaching progress by introducing the InternVL3 multimodal reasoning mechanism driven by the chain-of-thought of prompt text, combined with PPT content, group facial expressions (such as confusion, bewilderment, silence, etc.) and speech semantics.

[0052] In this embodiment, the combination of the current overall student status label and teaching progress label includes: [Status label: Focused, Teaching progress label: Normal], [Status label: Confused, Teaching progress label: Too fast], [Status label: Relaxed, Teaching progress label: Too slow], [Status label: Silent, Teaching progress label: Too fast] and [Status label: Silent, Teaching progress label: Too slow].

[0053] Furthermore, several new combinations can be set for the current overall student status labels and teaching progress labels, allowing for further refinement of the combinations and enabling the InternVL3 model to perform more detailed learning.

[0054] This invention decouples intervention actions into a combination and matching of the overall student status labels and teaching progress labels.

[0055] S3. Input the final input text into the pre-trained InternVL3 model in the cloud to obtain the final inference result, and push the final inference result to the teacher's terminal. The teacher can then provide teaching guidance based on the inference result.

[0056] This step implements part of the framework for cloud-based multimodal reasoning—combinatorial strategy execution.

[0057] S31. Input the final input text into the pre-trained InternVL3 model for inference to obtain a combination of the current overall student status label and teaching progress label.

[0058] S32. The pre-trained InternVL3 model outputs the final inference result based on the combination of the current overall student status label and the teaching progress label.

[0059] When the obtained combination is [state label: focused, teaching progress label: normal], the pre-trained InternVL3 model outputs the final inference result: the teaching progress is appropriate, the teacher can continue teaching and no adjustments are needed.

[0060] When the obtained combination is [Status label: Confused, Teaching progress label: Too fast], the pre-trained InternVL3 model outputs the final inference result: suggesting that the teacher should elaborate on the knowledge points being explained.

[0061] When the obtained combination is [Status label: Easy, Teaching progress label: Too slow], the pre-trained InternVL3 model outputs the final inference result: The students have mastered the current content, and it is recommended that the teacher speed up the pace of the explanation.

[0062] When the obtained combination is [Status label: Silent, Teaching progress label: Too fast] or [Status label: Silent, Teaching progress label: Too slow], the pre-trained InternVL3 model first outputs the structured fields of the proposed recommended questions. Then, based on the structured fields of the proposed recommended questions, the pre-trained InternVL3 model calls the recommended question acquisition module to obtain the question stem, options, and answers of the recommended questions. The pre-trained InternVL3 model outputs the obtained question stem, options, answers, and attached teaching suggestions as the final reasoning result.

[0063] In practice, when the obtained combination is [state label: silent, teaching progress label: too fast], the pre-trained InternVL3 model outputs recommended questions that are of medium difficulty, medium difficulty, medium difficulty, medium difficulty, difficult multiple choice questions, or difficult difficulty.

[0064] When the obtained combination is [state label: silent, teaching progress label: too slow], the pre-trained InternVL3 model outputs recommended questions that tend to be simple multiple-choice questions or simple calculation questions.

[0065] Furthermore, the recommended topics can be further adjusted based on subsequent feedback from teachers.

[0066] The structured fields include the subject category (Cat) of the proposed questions, the knowledge point name (Concept), the difficulty value (Diff) of the recommended questions, the question type (Qtype) (including multiple choice, calculation, and application questions), and the natural language question description (desc).

[0067] Furthermore, based on several new combinations, the output of the pre-trained InternVL3 model can be further refined to make the output of the InternVL3 model more refined and accurate.

[0068] The recommended question acquisition module is implemented according to the following steps: D1. The pre-trained Sentence-BERT model is used to perform semantic encoding on the structured fields and knowledge point names respectively, so as to obtain the semantic vectors of the structured fields and the semantic vectors of the knowledge points respectively.

[0069] The structured field semantic vector vdesc = SBERT(question description field desc), and the knowledge point semantic vector vconcept = SBERT(concept_name).

[0070] D2. Expand the recommended question difficulty values ​​into a one-dimensional vector.

[0071] D3. Concatenate the semantic vector, the knowledge point semantic vector, and the recommended question difficulty value of the one-dimensional vector in sequence to generate the final question requirement vector.

[0072] q = [vdesc; vconcept; diff], where q is the final question requirement vector, vdesc is the semantic vector, the knowledge point semantic vector is the knowledge point semantic vector, and diff is the difficulty value of the one-dimensional vector.

[0073] Furthermore, this question demand vector is used to perform similarity retrieval in the question bank, ensuring that the returned questions are aligned with teaching needs in three dimensions: semantic content, knowledge scope, and cognitive difficulty.

[0074] D4. Based on the preset structured question bank and question demand vector, obtain recommended questions, and obtain the question stem, options and answers of the recommended questions.

[0075] The process of obtaining recommended questions based on a pre-defined structured question bank and question requirement vectors is as follows: In the pre-defined structured question bank, each question has a pre-constructed corresponding embedding vector, constructed in the same way as the question requirement vector. Recommended questions are obtained by processing them using the following formula: in, Represents the vector required by the problem; For indexing; This indicates the first question in the pre-defined structured question bank. Embedding vectors corresponding to each question; In order to make Embedded vector reaches maximum The corresponding questions also indicate recommended questions; This indicates taking the maximum value; Indicates cosine similarity; This represents the total number of questions in the structured question bank. Table L2 norm; Represents the vector of requirements for the problem. The transpose of .

[0076] This invention decouples the intervention action into question bank vector retrieval. The two-stage process of "combined matching of overall student status labels and teaching progress labels + question bank vector retrieval" avoids the risks of knowledge errors and difficulty mismatch caused by directly generating questions from a large model.

[0077] The question bank retrieval is based on SBERT fusion embedding and structured metadata, and also ensures that the returned questions are highly matched with teaching needs in terms of semantics, knowledge scope and cognitive difficulty.

[0078] S33. The reasoning results are pushed to the classroom terminal, and the teacher provides teaching guidance based on the obtained reasoning results.

[0079] S4. After a preset time period, repeat steps S1-S3 to obtain new progress assessment and teaching support strategies, thereby updating the progress assessment and teaching support.

[0080] like Figure 2 As shown, this embodiment also provides a multimodal classroom progress assessment and teaching assistance system based on an edge-cloud collaborative architecture, including: The classroom-side data acquisition module synchronously collects the teacher's PPT slides, facial expression dataset, and text / speech data on the classroom end. The input text acquisition module concatenates all data collected by the classroom-side data acquisition module with preset prompt text to obtain the final input text. The model inference and teacher-side result acquisition module inputs the final input text into a pre-trained InternVL3 model for inference, obtains the inference result, and pushes the inference result to the teacher's terminal, allowing the teacher to provide teaching guidance based on the inference result.

[0081] This invention focuses on constructing a collaborative, edge-cloud-based classroom support framework with decoupled perception, decision-making, and execution: the edge is responsible for privacy-preserving multimodal data collection and lightweight encoding; the cloud utilizes multimodal large-scale models such as InternVL3 for chain-of-thought reasoning, outputting structured progress assessments and intervention intentions; then, through a pre-set combination pool and structured question bank, the abstract intentions are transformed into concrete, reliable, and traceable teaching actions (such as pushing questions of specified knowledge points and difficulty). This method not only achieves accurate perception of teaching progress but also ensures the professionalism and security of intervention content, effectively solving the key bottlenecks of "inaccurate perception and unreliable intervention" in existing technologies.

[0082] Specifically, this invention, at the multimodal perception layer, synchronously collects PPT slides, keyframes of student facial expressions, and Whisper transcribed text via edge devices, achieving lightweight data encoding while ensuring privacy. At the cloud decision layer, multi-source information is constructed into a unified Prompt input InternVL3 multimodal large model, forcing it to reason in a chain-of-thought manner, first analyzing content difficulty, explanation speed, and student reactions, and then outputting structured progress labels (too fast / too slow / normal) and intervention intentions. At the execution feedback layer, the system matches actions in a preset strategy pool based on progress labels and student performance states (such as confusion, bewilderment, silence, etc.), and generates structured question requirements containing knowledge points, target difficulty, question types, and natural language descriptions. Finally, the most matching question is returned from the question bank through vector retrieval, rather than relying on the large model for free generation, ensuring the professionalism and reliability of the content.

[0083] Example 2: Figure 2 This invention demonstrates a specific application scenario in a second-grade elementary school math class. The aim of this invention is to assess classroom progress in real time and provide teachers with precise, personalized support suggestions.

[0084] Scenario Description: In a second-grade math class, Teacher Li is explaining mixed operations of adding a two-digit number to a one-digit number (without carrying). The system of this invention collects the following data in real time at the classroom end: (1) Facial expression dataset: The current frame image, including the faces of all students, is obtained through a classroom camera array during class. The current frame image is used as the group expression analysis frame. Face detection is performed on the group expression analysis frame to obtain the detection box image of each student's facial expression. The detection box images of all students' facial expressions are summarized to obtain the facial expression dataset.

[0085] (2) PPT screen: Teacher Li’s current PPT page displays the example problem “23+5=?” and its solution steps.

[0086] (3) Classroom audio transcription: Teacher Li's audio was transcribed into text in real time: "Students, please look at this problem, 23 plus 5. We align the ones place, add the ones place to the ones place, 3 plus 5 equals 8..."

[0087] System workflow: Cloud-based multimodal data fusion and inference: Data fusion: The system merges the "facial expression dataset", "PPT screen" and "teacher's text and voice" obtained from the classroom end in the cloud to construct a multimodal context (input text).

[0088] InternVL3 Inference: Input the fused context into the InternVL3 large model.

[0089] Model comprehensive analysis: (1) Facial silence: Multiple students' facial expressions showed "silence". (2) PPT content: It was determined that the knowledge point being explained was "two-digit number plus one-digit number (without carrying)". (3) Text and voice information: It was confirmed that Teacher Li was explaining the calculation process of the example problem in detail.

[0090] Thought chain output: InternVL3 reasoning concludes: [Status label: Silent, Teaching progress label: Too fast], students have difficulty understanding the knowledge point 'adding the units digits'. The pre-trained InternVL3 model outputs structured fields for the proposed recommended questions based on the obtained [state label: silent, teaching progress label: too fast]. Subject category: Mathematics; Knowledge point name: Adding a two-digit number to a one-digit number (adding the units digits); Recommended difficulty level: Easy; Question type: Calculation problem; Natural Language Problem Description: Please complete a simple exercise involving adding a two-digit number to the units digit of a one-digit number.

[0091] The system performs semantic encoding on structured fields and knowledge point names, resulting in semantic vectors for structured fields and knowledge points, respectively. The recommended question difficulty value is expanded into a one-dimensional vector. Finally, the semantic vector, the knowledge point semantic vector, and the one-dimensional vector of recommended question difficulty value are concatenated to generate the final question requirement vector.

[0092] Question bank vector similarity retrieval: The system performs vector retrieval in the question bank to find the most matching question, for example: "Please calculate: 2+6=?".

[0093] Response Generation: The system recommends this question to Teacher Li and displays a prompt on the teacher's terminal's auxiliary interface: "It is recommended that the teacher slow down the pace and ask students to complete the exercise on the blackboard: 2+6=?". The system also includes brief teaching suggestions: "Guide students to reiterate the steps for adding the ones digits together."

[0094] Five seconds later, the system repeats the above steps, generates a new response, and pushes it to Teacher Li.

[0095] In contrast, traditional teaching methods may fail to detect such subtle student silences in real time, preventing Teacher Li from adjusting her teaching strategies promptly. Other tutoring methods based on large models may directly generate questions, while this system ensures the accuracy and relevance of questions to knowledge points through structured retrieval, avoiding the "illusion" problem that may occur with generative AI. By recommending specific practice questions and providing clear intervention suggestions, this system effectively guides students to actively think about and consolidate knowledge points, rather than passively receiving problem-solving processes.

[0096] Furthermore, this invention achieves a balance between low-latency response and high educational credibility by decoupling the "perception-reasoning-retrieval-feedback" process: only anonymized multimodal data is uploaded to the classroom end, meeting privacy compliance requirements; cloud-based reasoning results are interpretable and auditable; and question bank retrieval results are traceable and replaceable. This framework is not only suitable for primary and secondary school classrooms but can also be extended to large-scale teaching scenarios such as university lectures and vocational training.

[0097] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture, characterized in that, Includes the following steps: S1. Simultaneously collect the teacher's PPT slides, facial expression dataset, and text / voice data on the classroom end. S2. Combine all the data collected in step S1 with the preset prompt text to obtain the final input text; S3. Input the final input text into the pre-trained InternVL3 model for inference to obtain the final inference result. Push the final inference result to the teacher's terminal, and the teacher provides teaching guidance based on the inference result. S4. After a preset time period, repeat steps S1-S3, so that the teacher can continue to provide teaching guidance based on the new reasoning results.

2. The multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture as described in claim 1, characterized in that, Step S1 specifically involves: S11. Obtain the current frame image, including the faces of all students, during the class, and use the current frame image as the group expression analysis frame. S12. Perform face detection on the group expression analysis frames to obtain the detection box image of each student's face expression. The detection box images of all students' face expressions are summarized to obtain the face expression dataset. S13. Obtain the PPT screen currently being presented by the teacher during class; S14. Obtain the teacher's voice from the previous several seconds to the current moment, and convert the teacher's voice into text-to-speech.

3. The multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture according to claim 1, characterized in that, Step S2 specifically involves: S21. Concatenate all images, PPT slides, and text / speech data in the facial expression dataset to obtain the text information. S22. Combine the preset prompt text with the text information to obtain the final input text.

4. The multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture as described in claim 3, characterized in that: The preset prompt text is obtained by sequentially analyzing the detection box image of each student's facial expression, PPT screen and text voice in the input text information, and then outputting a combination of the current status label of the overall students and the teaching progress label. The combination of the current overall student status label and teaching progress label includes: [Status label: Focused, Teaching progress label: Normal], [Status label: Confused, Teaching progress label: Too fast], [Status label: Relaxed, Teaching progress label: Too slow], [Status label: Silent, Teaching progress label: Too fast] and [Status label: Silent, Teaching progress label: Too slow].

5. The multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture according to claim 1, characterized in that, Step S3 specifically involves: S31. Input the final input text into the pre-trained InternVL3 model for inference to obtain a combination of the current overall student status label and teaching progress label; S32. The pre-trained InternVL3 model outputs the final inference result based on the combination of the current overall student status label and the teaching progress label. S33. The reasoning results are pushed to the classroom terminal, and the teacher provides teaching guidance based on the obtained reasoning results.

6. The multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture according to claim 5, characterized in that, Step S32 specifically involves: When the obtained combination is [state label: focused, teaching progress label: normal], the pre-trained InternVL3 model outputs the final inference result: the teaching progress is appropriate, the teacher can continue teaching and no adjustments are needed; When the obtained combination is [Status label: Confused, Teaching progress label: Too fast], the pre-trained InternVL3 model outputs the final inference result: suggesting that the teacher should explain the knowledge points being taught in more detail. When the obtained combination is [Status label: Easy, Teaching progress label: Too slow], the pre-trained InternVL3 model outputs the final inference result: The students have mastered the current content, and it is recommended that the teacher speed up the pace of the explanation; When the obtained combination is [Status label: Silent, Teaching progress label: Too fast] or [Status label: Silent, Teaching progress label: Too slow], the pre-trained InternVL3 model first outputs the structured fields of the proposed recommended questions. Then, based on the structured fields of the proposed recommended questions, the pre-trained InternVL3 model calls the recommended question acquisition module to obtain the question stem, options, and answers of the recommended questions. The pre-trained InternVL3 model outputs the obtained question stem, options, answers, and attached teaching suggestions as the final reasoning result.

7. The multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture as described in claim 6, characterized in that: The structured fields include the subject category of the recommended questions, the name of the knowledge point, the difficulty value of the recommended questions, the question type, and the natural language description of the questions.

8. The multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture according to claim 6, characterized in that, The recommended question acquisition module is implemented according to the following steps: D1. The pre-trained Sentence-BERT model is used to perform semantic encoding on the structured fields and knowledge point names respectively, so as to obtain the semantic vectors of the structured fields and the semantic vectors of the knowledge points respectively. D2. Expand the recommended question difficulty values ​​into a one-dimensional vector; D3. Concatenate the recommended question difficulty values ​​of the semantic vector, knowledge point semantic vector, and one-dimensional vector in sequence to generate the final question requirement vector. D4. Based on the preset structured question bank and question demand vector, obtain recommended questions, and obtain the question stem, options and answers of the recommended questions.

9. A multimodal classroom progress assessment and teaching assistance method based on an edge-cloud collaborative architecture as described in claim 8, characterized in that, The specific steps for obtaining recommended questions based on a pre-set structured question bank and question demand vector are as follows: In the pre-set structured question bank, each question has a pre-constructed corresponding embedding vector, which is constructed in the same way as the question requirement vector. The recommended questions are obtained by processing them according to the following formula: in, Represents the vector required by the problem; For indexing; This indicates the first question in the pre-defined structured question bank. Embedding vectors corresponding to each question; In order to make Embedded vector reaches maximum The corresponding questions also indicate recommended questions; This indicates taking the maximum value; Indicates cosine similarity; This represents the total number of questions in the structured question bank. Table L2 norm; Represents the vector of requirements for the problem. The transpose of .

10. A system employing any one of the edge-cloud collaborative architecture-based multimodal classroom progress assessment and teaching assistance methods as described in claims 1-9, characterized in that, include: The classroom-side data acquisition module synchronously collects the teacher's PPT slides, facial expression datasets, and the teacher's text and voice at the classroom end. The input text acquisition module combines all the data collected by the classroom-side data acquisition module with the preset prompt text to obtain the final input text. The model inference and teacher-side result acquisition module inputs the final input text into the pre-trained InternVL3 model to obtain the inference result, and pushes the inference result to the teacher's terminal, so that the teacher can provide teaching guidance based on the inference result.

Images