A classroom evaluation method based on multi-modal data and related equipment

Abstract

The embodiment of the application discloses a classroom evaluation method based on multi-modal data and related equipment, which comprehensively and objectively records the classroom teaching process by identifying the number of times and time of teacher inspection, the content and number of times of writing on the blackboard, the content of teaching PPT, the teaching elements such as speech speed, and combining various behavior data such as student expression, head raising rate and interaction times. On this basis, the first preset evaluation rule is used to quantitatively analyze each behavior data, and an evaluation score is automatically generated, thereby avoiding the standard deviation and low efficiency of human evaluation, and improving the real-time and accuracy of classroom teaching evaluation.

Description

Technical Field

[0001] This invention relates to the field of educational data processing technology, and in particular to a classroom evaluation method and related equipment based on multimodal data. Background Technology

[0002] With the deepening of educational informatization, classroom teaching evaluation has gradually become a key link in improving teaching quality. However, the traditional lesson observation and evaluation model mainly relies on manual observation and subjective experience, which has problems such as inconsistent evaluation standards, limited coverage, and delayed feedback. Moreover, it lacks objective, real-time, and quantitative data support, resulting in inaccurate scoring. Summary of the Invention

[0003] In view of this, the present invention provides a classroom assessment method and related equipment based on multimodal data.

[0004] The specific technical solution of the first embodiment of the present invention is as follows: a classroom evaluation method based on multimodal data, the method comprising: acquiring classroom teaching videos of teachers and class status videos of students during the same time period; extracting teacher behavior and teaching content from the classroom teaching videos, and extracting student behavior from the class status videos; the teacher behavior includes the number of times the teacher walks around the podium, the time spent walking around, and the number of times the teacher writes on the blackboard; the teaching content includes the content of the blackboard writing, the content of the teaching PPT, and the average speaking speed; the student behavior includes student facial expressions, the student head-up rate, and the number of interactions; and quantifying the teacher behavior, the teaching content, and the student behavior based on a first preset evaluation rule to obtain an evaluation score for the current class.

[0005] Preferably, the average speaking speed is obtained by the following method: extracting the audio file of the classroom teaching video and obtaining the corresponding classroom transcript text; extracting the speech duration of the teacher in the audio file; and obtaining the average speaking speed based on the speech duration and the total number of words in the classroom transcript text.

[0006] Preferably, the number of times the teacher patrols off the podium and the patrol time are obtained using the following method: identifying the teacher's podium activity range and the student seating range in the classroom teaching video; assigning a label to the teacher in the classroom teaching video; obtaining the continuous movement trajectory of the label in the classroom teaching video; obtaining the number of times the teacher patrols off the podium based on the continuous movement trajectory, the teacher's podium activity range, the student seating range, and a preset patrol number evaluation rule; the preset patrol number evaluation rule includes recording one patrol when the continuous movement trajectory moves out of the teacher's podium activity range and directly enters the student seating range; obtaining the first time corresponding to the continuous movement trajectory being within the student seating range, where the first time is the patrol time.

[0007] Preferably, the step of quantitatively evaluating the teacher's behavior, the teaching content, and the student's behavior based on a first preset evaluation rule to obtain an evaluation score for the current lesson includes: constructing a classroom focus curve for the current lesson based on student facial expressions and student head-up rate, and obtaining a first teaching score for the classroom focus curve based on a preset focus scoring rule for the classroom focus curve; scoring the teaching content of the blackboard writing and teaching PPT content using a preset content evaluation model to obtain a second teaching score for the blackboard writing and a third teaching score for the PPT content; constructing a knowledge graph based on the classroom transcript text corresponding to the classroom teaching video; obtaining a fourth teaching score based on the number of times the teacher walks around the podium, the time spent walking around the podium, the number of times the teacher writes on the blackboard, the average speaking speed during the lecture, the number of interactions, the knowledge graph, and the second preset evaluation rule; and weighting the first teaching score, the second teaching score, the third teaching score, and the fourth teaching score to obtain the final evaluation score.

[0008] Preferably, the step of constructing the classroom focus curve for this class based on the student's facial expressions and the student's head-up rate includes: obtaining the types of each student's facial expressions at different times, the types including focus and boredom; calculating the proportion of all students' facial expressions belonging to focus at different times; weighting the proportion and the student's head-up rate to obtain the instantaneous group attention intensity at each time; and constructing the classroom focus curve based on the instantaneous group attention intensity at all times.

[0009] Preferably, obtaining the first teaching score of the classroom focus curve according to the preset focus scoring rules of the classroom focus curve includes: obtaining attention troughs in the classroom focus curve, wherein the attention troughs are different time periods consisting of consecutive moments when the instantaneous group attention intensity is lower than a first preset threshold; obtaining attention recovery periods based on the attention troughs, the first preset threshold, and a second preset threshold; wherein the attention recovery period is the time period from the end of each attention trough to the moment when the instantaneous group attention intensity first reaches a second preset threshold; wherein the second preset threshold is greater than the first preset threshold; obtaining the first instantaneous group attention intensity average and the first total duration of all attention troughs; obtaining the second instantaneous group attention intensity average and the second total duration of all attention recovery periods; obtaining an attention resilience index based on the first instantaneous group attention intensity average, the first total duration, the second instantaneous group attention intensity average, and the second total duration; wherein the attention resilience index is used to characterize the average attention level of students in this class; and obtaining the first teaching score of the classroom focus curve based on the attention resilience index and the preset attention scoring rules.

[0010] Preferably, the attention resilience index is obtained using the following formula:

[0011] in, The attention resilience index, This represents the average group attention intensity at the second instant. The average value of the group's attention intensity at the first instant. This is the first total duration. This is the second total duration.

[0012] The specific technical solution of the second embodiment of the present invention is as follows: a classroom evaluation system based on multimodal data, the system comprising: a video acquisition module, a content extraction module, and a scoring module; the video acquisition module is used to acquire classroom teaching videos of teachers and student class status videos during the same time period; the content extraction module is used to extract teacher behavior and teaching content from the classroom teaching videos, and extract student behavior from the class status videos; the teacher behavior includes the number of times they walk around the podium, the time of their walk, and the number of times they write on the blackboard; the teaching content includes blackboard content, teaching PPT content, and average speaking speed; the student behavior includes student facial expressions, student head-up rate, and number of interactions; the scoring module is used to quantitatively evaluate the teacher behavior, the teaching content, and the student behavior based on a first preset evaluation rule to obtain an evaluation score for the current class.

[0013] The specific technical solution of the third embodiment of the present invention is as follows: a classroom assessment device based on multimodal data, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method as described in any one of the first embodiments of this application.

[0014] The specific technical solution of the fourth embodiment of the present invention is as follows: a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor performs the steps of the method as described in any one of the first embodiments of this application.

[0015] Implementing the embodiments of the present invention will have the following beneficial effects: This invention achieves a comprehensive and objective record of the classroom teaching process by identifying teaching elements such as the number and time of teacher patrols, blackboard content and number of times writing on the blackboard, teaching PPT content, and speaking speed, combined with various behavioral data such as student facial expressions, head-up rate, and number of interactions. Based on this, it quantifies and analyzes various behavioral data according to a first preset evaluation rule, automatically generating evaluation scores, avoiding the standard deviation and inefficiency of human evaluation, thereby improving the real-time nature and accuracy of classroom teaching evaluation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the steps of a classroom assessment method based on multimodal data. Figure 2 A schematic diagram of classroom attention curves; Figure 3 This is a schematic diagram of the structure of a classroom assessment system based on multimodal data; Among them, 201 is the video acquisition module; 202 is the content extraction module; and 203 is the rating module. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such processes, methods, products, or apparatus.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Please see Figure 1The above is a flowchart illustrating the steps of a classroom evaluation method based on multimodal data in the first embodiment of this application, aimed at improving the real-time performance and accuracy of classroom teaching evaluation. The method includes: Step 101: Obtain classroom teaching videos of teachers and students' classroom status during the same time period; Step 102: Extract teacher behavior and teaching content from the classroom teaching video, and extract student behavior from the class status video; the teacher behavior includes the number of times the teacher walks around the podium, the time of walking around, and the number of times the teacher writes on the blackboard; the teaching content includes the content of the blackboard writing, the content of the teaching PPT, and the average speaking speed; the student behavior includes student facial expressions, student head-up rate, and the number of interactions. Step 103: Quantitatively evaluate the teacher's behavior, the teaching content, and the student's behavior based on the first preset evaluation rules to obtain the evaluation score for this class.

[0022] Specifically, within the same class period, a 45-minute video of the teacher's classroom teaching and a video of the students' classroom status are simultaneously collected. Alternatively, a panoramic teaching video is collected using a panoramic AI camera at the back of the classroom, including both the teacher's and students' classroom status videos. Teacher behavior is extracted from the classroom teaching video: for example, the teacher walks around the podium 3 times during the class, totaling approximately 15 minutes, and writes on the blackboard 8 times. Simultaneously, teaching content is extracted: OCR recognition is performed on the blackboard area to obtain the text; keyframes and content of the displayed teaching PPT are captured using screen content recognition technology; and the teacher's average speaking speed is calculated to be 200 words per minute using speech recognition and timing. Student behavior is extracted from the classroom status video: facial expression recognition technology is used to statistically analyze the distribution ratio of "focused" or "bored" student expressions; head posture estimation is used to calculate the average student head-up rate for the entire class to be 85%; and based on voice activity and hand-raising gesture recognition, the number of student interactions, including group answers and individual questions, is recorded as 12 times. The large model is used to comprehensively evaluate the above behavioral data. For example, the number of times students patrol the podium, the patrol time, the number of times they write on the blackboard, the number of interactions, the student head-up rate, and the student facial expressions are used as positive weighting items, while excessive speaking speed and missing key content on the blackboard or PPT are used as negative adjustment items. The final output is an evaluation score of 92 points (out of 100) for this class teaching. The scores of each item and brief comments can be displayed on the interface, thus realizing a real-time and objective quantitative evaluation of the quality of classroom teaching.

[0023] The method in this embodiment identifies teaching elements such as the number and time of teacher patrols, blackboard content and number of times writing on the blackboard, teaching PPT content, and speaking speed. Combined with various behavioral data such as student facial expressions, head-up rate, and number of interactions, it achieves a comprehensive and objective record of the classroom teaching process. Based on this, it performs quantitative analysis on various behavioral data according to the first preset evaluation rules and automatically generates evaluation scores, avoiding the standard deviation and inefficiency of human evaluation, thereby improving the real-time and accuracy of classroom teaching evaluation.

[0024] In a specific embodiment, the average speaking speed of the lecture is obtained by the following method: extracting the audio file of the classroom teaching video and obtaining the classroom transcript text corresponding to the audio file; extracting the speech duration of the teacher's speech in the audio file; and obtaining the average speaking speed of the lecture based on the speech duration and the total number of words corresponding to the classroom transcript text.

[0025] Specifically, the audio file was extracted from the classroom teaching video. Using speech recognition technology, the entire audio segment was converted into a transcript of the classroom session, yielding approximately 9,000 words of text representing the teacher's speech. Voiceprint recognition and silence detection technologies were then used to accurately extract speech segments from the audio, and the pure speech duration, after removing student responses, classroom silence, and background noise, was calculated to be 40 minutes. Finally, based on the speech duration and the total number of words in the transcript, the average speaking speed for the lesson was automatically calculated as: 9,000 words / 45 minutes = 200 words / minute. By recognizing and distinguishing the teacher's voice and filtering out non-speaking silence periods, the purity and accuracy of the effective speaking speed calculation were ensured, avoiding errors caused by including classroom silence and student interaction time in the time calculation.

[0026] In a specific embodiment, the number of times the teacher patrols off the podium and the patrol time are obtained using the following method: identifying the teacher's podium activity range and the student seating range in the classroom teaching video; assigning a label to the teacher in the classroom teaching video; obtaining the continuous movement trajectory of the label in the classroom teaching video; obtaining the number of times the teacher patrols off the podium based on the continuous movement trajectory, the teacher's podium activity range, the student seating range, and a preset patrol number evaluation rule; the preset patrol number evaluation rule includes recording one patrol when the continuous movement trajectory moves out of the teacher's podium activity range and directly enters the student seating range; obtaining the first time corresponding to the continuous movement trajectory being within the student seating range, where the first time is the patrol time.

[0027] Specifically, a scene understanding model is used to identify and define the teacher's activity area at the podium (typically including the podium, blackboard, and multimedia operation area) and the student seating area (the classroom seating area) within the video frame. The teacher's head is set as a key point, and a multi-target tracking algorithm is used to acquire the continuous movement trajectory of the key point in the video in real time. The trajectory is analyzed according to a preset rule for judging the number of patrols: when the teacher's continuous movement trajectory completely moves out of the podium activity area, and its subsequent movement point directly enters the student seating area, it is determined that a valid podium patrol has occurred. Specifically, in this lesson, three such events were recorded according to this rule, i.e., the number of podium patrols was 3. At the same time, the time corresponding to the teacher's movement trajectory being within the student seating area can be accurately calculated, and these are added together to obtain the total patrol time of the teacher in this lesson, which is approximately 15 minutes. By locating the teacher's position using the head key point, the movement trajectory can be determined using a small number of key points, which can reduce the system's computing resources and time required for processing video data, and improve the calculation efficiency of the number of podium patrols and patrol time. By clearly defining what constitutes an effective inspection through pre-set rules, interference from teachers pacing around the podium or heading to the equipment area is effectively eliminated.

[0028] In a specific embodiment, the step of quantitatively evaluating the teacher's behavior, the teaching content, and the student's behavior based on a first preset evaluation rule to obtain an evaluation score for the current class includes: constructing a classroom focus curve for the current class based on student facial expressions and student head-up rate, and obtaining a first teaching score for the classroom focus curve based on a preset focus scoring rule for the classroom focus curve; scoring the teaching content of the blackboard writing and teaching PPT content using a preset content evaluation model to obtain a second teaching score for the blackboard writing and a third teaching score for the PPT content; constructing a knowledge graph based on the classroom transcript text corresponding to the classroom teaching video; obtaining a fourth teaching score based on the number of times the teacher walks around the podium, the time spent walking around the podium, the number of times the teacher writes on the blackboard, the average speaking speed during the lecture, the number of interactions, the knowledge graph, and the second preset evaluation rule; and weighting the first teaching score, the second teaching score, the third teaching score, and the fourth teaching score to obtain the final evaluation score.

[0029] Specifically, the overall classroom atmosphere and engagement were assessed: Based on real-time statistics of student facial expressions and head-up rates, a classroom attention curve was constructed with time as the horizontal axis. The first teaching score corresponding to the classroom attention curve was calculated as 28 points (out of 30) according to the pre-set attention scoring rules. The quality of the teaching content presentation was assessed: Using a pre-set content evaluation model, the extracted blackboard writing and teaching PPT content were evaluated separately. The model scored based on factors such as the logicality and emphasis of key points, resulting in a second teaching score of 18 points (out of 20) for the blackboard writing and a third teaching score of 19 points (out of 20) for the PPT content. The teaching process and interaction were assessed: First, based on the classroom transcript corresponding to the classroom teaching video, a knowledge graph for this lesson was constructed using natural language processing technology to quantify the completeness and logic of the knowledge structure. Subsequently, combining the number of times the teacher patrolled the podium (3 times), the patrol time (15 minutes), the number of times the teacher wrote on the blackboard (8 times), the average speaking speed (200 words / minute), the number of interactions (12 times), and the indicators of the knowledge graph, the fourth teaching score was calculated based on the second preset evaluation rules (such as whether the patrol time met the standard, whether the speaking speed was appropriate, and the richness of knowledge graph nodes) according to the preset large model. The score was 27 out of 30. The first teaching score (28), the second teaching score (18), the third teaching score (19), and the fourth teaching score (27) were added together to calculate the final evaluation score of the class, which was 92. This method not only focuses on the teacher's behavior and the students' status, but also incorporates the teaching content itself (blackboard writing, PPT, knowledge graph) into the evaluation system. It deconstructs the evaluation from three levels: attractiveness, presentation quality, and process interaction, making the evaluation results more comprehensive, three-dimensional, and closer to the essence of teaching.

[0030] Specifically, the pre-defined content evaluation model and the rules for evaluating teaching content can be found in the table below: Table 1: Content Evaluation Rules Table

[0031] In a specific embodiment, constructing the classroom focus curve for this class based on the student's facial expressions and the student's head-up rate includes: obtaining the types of each student's facial expressions at different times, the types including focus and boredom; calculating the proportion of all students' facial expressions belonging to focus at different times; weighting the proportion and the student's head-up rate to obtain the instantaneous group attention intensity at each time; and constructing the classroom focus curve based on the instantaneous group attention intensity at all times.

[0032] Specifically, after acquiring video footage of students' classroom behavior, image recognition technology is used to accurately identify each student's facial expressions at different times, determining whether the expression is focused or bored. For example, five minutes into the class, image recognition analysis reveals that out of 40 students, 32 are focused and 8 are bored. Based on this, the proportion of students with focused expressions at that moment is calculated to be 32 ÷ 40 = 80%. Simultaneously, the student head-up rate at that moment is also obtained, for example, 85%. The proportion of focused expressions and the student head-up rate are weighted according to preset weights. If the weight for the focused expression proportion is set to 0.6 and the weight for the student head-up rate is set to 0.4, then the instantaneous group attention intensity at that moment is 80% × 0.6 + 85% × 0.4 = 82%. Following this method, the instantaneous group attention intensity is calculated for each moment of the class. Finally, based on the instantaneous group attention intensity data for all moments, a classroom focus curve for this lesson is plotted. Please refer to the diagram for a schematic of the classroom focus curve. Figure 2 It visually presents changes in students' attention levels in class.

[0033] In a specific embodiment, obtaining the first teaching score of the classroom focus curve according to the preset focus scoring rules of the classroom focus curve includes: obtaining attention troughs in the classroom focus curve, wherein the attention troughs are different time periods consisting of consecutive moments when the instantaneous group attention intensity is lower than a first preset threshold; obtaining attention recovery periods based on the attention troughs, the first preset threshold, and a second preset threshold; wherein the attention recovery period is the time period from the end of each attention trough to the moment when the instantaneous group attention intensity first reaches a second preset threshold; wherein the second preset threshold is greater than the first preset threshold; obtaining the first instantaneous group attention intensity average value and the first total duration of all attention troughs; obtaining the second instantaneous group attention intensity average value and the second total duration of all attention recovery periods; obtaining an attention resilience index based on the first instantaneous group attention intensity average value, the first total duration, the second instantaneous group attention intensity average value, and the second total duration; wherein the attention resilience index is used to characterize the average attention level of students in this class; and obtaining the first teaching score of the classroom focus curve based on the attention resilience index and the preset attention scoring rules.

[0034] Specifically, the first and second preset thresholds can be set according to actual conditions. For example, the first preset threshold can be 80%, and the second preset threshold can be 83%. Figure 2It is known that there are two periods of attention troughs where the instantaneous group attention intensity is below 80%: approximately from the 5th to the 15th minute and from the 33rd to the 45th minute. The first period lasts for approximately 22 minutes, with an average instantaneous group attention intensity of about 77%. After the 15th minute, until the 17th minute, the instantaneous group attention intensity reaches 83%, representing a 2-minute attention recovery period with an average intensity of about 82%. This pattern continues, calculating the total time and average intensity of subsequent attention recovery periods. This method treats changes in student attention as a dynamic process, setting clear first and second preset thresholds and allowing for reasonable fluctuations in attention. However, it focuses on whether attention falls below the first preset threshold and whether it can effectively recover to a better state (the second preset threshold). Through clearly defined troughs and recovery periods, abstract attention management is transformed into measurable and calculable time periods and intensity values, forming an attention resilience index.

[0035] In a specific embodiment, the attention resilience index is obtained using the following formula:

[0036] in, The attention resilience index, This represents the average group attention intensity at the second instant. The average value of the group's attention intensity at the first instant. This is the first total duration. This is the second total duration.

[0037] Specifically, molecules ( This directly measures the extent to which attention recovers from a low point to the second preset threshold, i.e., the degree of attention improvement brought about by the teaching intervention. The denominator contains... As a penalty, the longer the slump lasts, the lower the resilience index. This prompts teaching evaluation to focus not only on "whether recovery is possible" but also on "how quickly it can be detected and intervened," encouraging teachers to address declining attention promptly. (Denominator) ) refers to recovery time The penalty term is used, but the growth of the logarithmic function is made to have a diminishing marginal effect. This means that the efficiency improvement of short recovery time contributes significantly to the exponent, while the penalty caused by a long recovery process will gradually slow down. This is in line with the actual teaching situation: it is an efficient performance for teachers to spend a few minutes to successfully improve their attention, but if it takes a very long time, such as half a class period to barely recover, the overall teaching efficiency has been greatly reduced.

[0038] In a specific embodiment, please refer to Figure 3This is a schematic diagram of a classroom evaluation system based on multimodal data according to the second embodiment of this application. The system includes: a video acquisition module 201, a content extraction module 202, and a scoring module 203. The video acquisition module 201 is used to acquire classroom teaching videos of teachers and student class status videos during the same time period. The content extraction module 202 is used to extract teacher behavior and teaching content from the classroom teaching videos, and to extract student behavior from the class status videos. The teacher behavior includes the number of times they walk around the podium, the time they spend walking around, and the number of times they write on the blackboard. The teaching content includes blackboard content, teaching PPT content, and average speaking speed. The student behavior includes student facial expressions, student head-up rate, and number of interactions. The scoring module 203 is used to quantitatively evaluate the teacher behavior, the teaching content, and the student behavior based on a first preset evaluation rule to obtain an evaluation score for this class.

[0039] The system in this embodiment identifies teaching elements such as the number and time of teacher patrols, blackboard content and number of times writing on the blackboard, teaching PPT content, and speaking speed. Combined with various behavioral data such as student facial expressions, head-up rate, and number of interactions, it achieves a comprehensive and objective record of the classroom teaching process. Based on this, it performs quantitative analysis on various behavioral data according to the first preset evaluation rules and automatically generates evaluation scores, avoiding the standard deviation and inefficiency of human evaluation, thereby improving the real-time performance and accuracy of classroom teaching evaluation.

[0040] In a specific embodiment, the third embodiment of this application provides a classroom assessment device based on multimodal data, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method as described in any of the first embodiments of this application.

[0041] Specifically, classroom assessment equipment based on multimodal data may also include the following modules: 1. Multimodal data acquisition module: Data Acquisition Terminal: Integrates offline equipment such as AI cameras, microphones, recording terminals, and microphones, as well as online platforms such as operating systems and teaching platforms, to form an "offline + online" data acquisition matrix; The data collected covers five main categories: teacher behavior (circling, blackboard writing, speaking speed), student status (facial expressions, head-up rate, interactive response), teaching content (PPT, blackboard writing, voice), and process-based learning data (pre-class records, homework mistakes). Data collection granularity: refined to the second level for behavioral data and semantic level for content data, such as the duration of students' hesitation when answering questions and the semantic classification of interactions.

[0042] 2. AI Evaluation Model Module: Teaching Situation Analysis Sub-model: Based on NLP technology, it identifies teachers' speaking speed (standard 200-250 words / minute), semantic logic, and sensitive sentences, with a false judgment rate of <3%; it integrates three major educational theory models: McCarthy 4MAT, Bloom's Taxonomy of Cognition, and Flanders Interaction Analysis, and quantifies indicators such as the proportion of question types and cognitive levels. Learning analysis sub-model: Through computer vision, various types of student behaviors are identified, attention curves are generated, and knowledge graphs are constructed by combining homework data to accurately locate weak knowledge points; Specialized evaluation sub-model: The ideological and political education of the course is evaluated from three dimensions: "teaching objectives-content-expression", and a score of ≥2.5 is qualified; in the "two-dimensional and one-dimensional" evaluation, the higher-level requirement is that the content of analysis / evaluation / creation accounts for ≥30%, the innovation requirement is that the content of interdisciplinary cases and AI-assisted teaching accounts for ≥20%, and the challenge requirement is that the time for in-depth thinking is ≥15 minutes.

[0043] 3. Multi-role application module: For teachers: The system outputs a "classroom profile + improvement checklist," along with comparisons to excellent lesson examples, resulting in a 60% increase in improvement efficiency. Key performance indicators include at least 3 effective questions per 10 minutes and at least 70% student engagement. Supervisory side: Supports simultaneous classroom observation in 16 classrooms, AI automatically marks anomalies, early warning response time <3 minutes, improving work efficiency by 70%, and requires AI and supervisor scoring consistency ICC value ≥0.98; Management side: Provides a "Teaching Quality Dashboard" to display macro data such as the rate of excellent courses in departments, the rate of ideological and political integration, and the average daily usage time of classrooms ≥4 class hours, to support resource optimization; On the student side: Based on data such as the attribution of homework errors and the response time of classroom interactions, personalized learning gaps can be identified and appropriate resources can be pushed.

[0044] 4. Cross-school data circulation module: Data anonymization: High-quality teaching data is anonymized to remove sensitive identity information while retaining behavioral characteristics and teaching patterns. Circulation mechanism design: Connect to the regional education data exchange and adopt a pay-per-use or authorized usage model; Balanced empowerment mechanism: After weaker universities reuse the model of key universities, the pass rate of ideological and political education can be increased from 35% to 72%.

[0045] Beneficial effects: Precise evaluation: Through multimodal data fusion and AI model analysis, teaching indicators are quantitatively evaluated, the consistency of evaluation results is improved to over 98%, and the accuracy rate of identifying higher-order thinking problems is ≥88%; Real-time feedback: Breaking away from the traditional 1-2 week evaluation cycle, an evaluation report with improvement suggestions is generated within 30 minutes, allowing teachers to adjust their teaching strategies in real time, resulting in an average increase of 45% in classroom participation; Quantification of special indicators: For the first time, a systematic and quantitative evaluation of core requirements such as ideological and political education in courses and "two genders and one degree" has been achieved, solving the pain point that 85% of universities have no available tools for special evaluation. Data security and controllability: Through technologies such as three-level rights confirmation and privacy computing, value is released while ensuring data security and compliance, and adapting to the requirements of the domestic environment; Empowering balanced education: Cross-school data flow mechanism narrows regional education gaps, improves management efficiency by more than 50%, and expands the coverage of teaching supervision from 20 lessons per week to 100 lessons; Lightweight deployment: It supports the combination of simulated data testing and small-scale real-world scenario verification, and the replication and deployment cycle is controlled within 15 working days. It is compatible with existing recording and monitoring equipment, reducing the construction cost of schools.

[0046] (I) System Architecture Deployment Hardware layer: Deploy AI cameras, microphones, recording terminals and other acquisition devices, adapt to protocols such as RTSP, RTMP, and ONVIF, and can be connected to the school's existing monitoring and recording equipment; Data layer: Construct a data storage architecture of "local + cloud backup", divide storage areas according to sensitivity level, use local encrypted storage for sensitive data, and share general data in the cloud; Algorithm layer: Integrates NLP language analysis algorithm, computer vision behavior recognition algorithm, and fusion evaluation algorithm based on educational theory, deployed on AI computing server, supports edge computing mode to reduce network bandwidth consumption; Application layer: Develop multi-role access points for web and mobile terminals, supporting functions such as four-screen / nine-screen / sixteen-screen simultaneous classroom observation, voice intercom, and early warning push notifications.

[0047] (II) Work Process Data collection phase: Real-time capture of classroom audio and video, teacher and student behavior data through offline terminals, and simultaneous acquisition of pre-class preparation, post-class homework, and other data through online platforms, forming a closed-loop data system covering the entire cycle of "pre-class - in-class - post-class". Data governance phase: The collected data is cleaned, labeled and verified. Sensitive data is processed using differential privacy. Important data is registered as an asset and timestamps and speaker tags are added in a standardized format. Model evaluation phase: The teaching situation analysis sub-model identifies teachers' speaking speed, teaching behavior and interaction quality; the learning situation analysis sub-model generates student attention curves and knowledge graphs; and the special evaluation sub-model completes the quantitative scoring of curriculum ideological and political education and "two genders and one degree". Results output phase: Deliver customized reports to teachers, supervisors, administrators, and students, including classroom profiles, improvement lists, quality dashboards, and personalized learning suggestions; Data circulation stage: High-quality data after anonymization is connected to the education data exchange, and cross-school sharing and reuse are achieved through an authorization mechanism.

[0048] (III) Pilot Verification Plan Tiered pilot program: First, cover 5-8 disciplines in 2-3 different types of universities, focusing on the evaluation of experimental courses in science and engineering and the evaluation of ideological and political education in humanities, and form discipline-adapted templates; Indicator Verification: Focus on verifying the achievement of core indicators, including a 20-30% increase in the course excellence rate, an ICC value ≥ 0.98 for the consistency between AI evaluation and human scoring, and a data processing delay ≤ 30 minutes; Promotion and optimization: Based on the feedback from pilot projects, iterate the model algorithm to form a standardized deployment process and operation specifications to support the rapid implementation of different types of universities.

[0049] In a specific embodiment, the fourth embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method as described in any one of the first embodiments of this application.

[0050] The above embodiments merely illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for class evaluation based on multi-modal data, characterized in that, The method includes: Obtain videos of teachers' classroom teaching and students' classroom behavior during the same time period; Extract teacher behavior and teaching content from the classroom teaching video, and extract student behavior from the class status video; the teacher behavior includes the number of times they walk around the podium, the time they spend walking around, and the number of times they write on the blackboard; the teaching content includes the content of the blackboard writing, the content of the teaching PPT, and the average speaking speed; the student behavior includes student facial expressions, student head-up rate, and the number of interactions. The teacher's behavior, the teaching content, and the student's behavior are quantitatively evaluated based on the first preset evaluation rules to obtain the evaluation score for this class.

2. The multi-modal data based classroom evaluation method of claim 1, wherein, The average speaking speed during the lecture was obtained using the following method: Extract the audio file from the classroom teaching video and obtain the corresponding classroom transcript text; Extract the duration of the teacher's speech from the audio file; The average speaking speed is obtained based on the duration of the audio and the total number of words in the classroom transcript.

3. The classroom assessment method based on multimodal data as described in claim 1, characterized in that, The number of times and duration of the lectern patrols were obtained using the following method: Identify the teacher's podium activity range and the students' seating range in the classroom teaching video; Tag the teachers in the classroom teaching videos; Obtain the continuous movement trajectory of the tag in the classroom teaching video; The number of times the teacher patrols off the platform is obtained based on the continuous movement trajectory, the teacher's activity range at the podium, the student seating area, and a preset patrol frequency evaluation rule; the preset patrol frequency evaluation rule includes... When the continuous movement trajectory moves out of the teacher's podium activity range and directly enters the student's seating area, it is recorded as one patrol of the podium. The first time corresponding to the continuous movement trajectory being within the student's seating area is obtained, and the first time is the patrol time.

4. The classroom assessment method based on multimodal data as described in claim 1, characterized in that, The quantitative evaluation of the teacher's behavior, the teaching content, and the student's behavior based on the first preset evaluation rules to obtain an evaluation score for this lesson includes: A classroom attention curve for this lesson is constructed based on the students' facial expressions and head-up rate, and a first teaching score for the classroom attention curve is obtained according to the preset attention scoring rules of the classroom attention curve. The teaching content of the blackboard writing and the teaching PPT is scored using a preset content evaluation model to obtain a second teaching score for the blackboard writing and a third teaching score for the PPT. A knowledge graph is constructed based on the classroom transcript text corresponding to the classroom teaching video; A fourth teaching score is obtained based on the number of times the student visits the podium, the time spent visiting the podium, the number of times the student writes on the blackboard, the average speaking speed during the lecture, the number of interactions, the knowledge graph, and the second preset evaluation rules. The evaluation score is obtained by weighting the first teaching score, the second teaching score, the third teaching score, and the fourth teaching score.

5. The classroom assessment method based on multimodal data as described in claim 4, characterized in that, The process of constructing a classroom attention curve based on student facial expressions and head-up rate includes: Acquire the types of each student's facial expressions at different times, including those of focus and boredom; Calculate the percentage of students whose facial expressions at different times were focused; The instantaneous group attention intensity at each moment is obtained by weighting the ratio and the student head-up rate. The classroom focus curve is constructed based on the instantaneous group attention intensity at all times.

6. The classroom assessment method based on multimodal data as described in claim 4, characterized in that, The step of obtaining the first teaching score of the classroom attention curve according to the preset attention scoring rules of the classroom attention curve includes: The attention troughs in the classroom focus curve are obtained, and the attention troughs are different time periods consisting of consecutive moments when the instantaneous group attention intensity is lower than a first preset threshold. An attention recovery period is obtained based on the attention trough period, the first preset threshold, and the second preset threshold; the attention recovery period is the time period from the end of each attention trough period to the moment when the instantaneous group attention intensity first reaches the second preset threshold; the second preset threshold is greater than the first preset threshold. Obtain the average group attention intensity and the first total duration of all attention troughs; Obtain the average group attention intensity at the second instant and the second total duration during all attention recovery periods; An attention resilience index is obtained based on the first instantaneous average group attention intensity, the first total duration, the second instantaneous average group attention intensity, and the second total duration; the attention resilience index is used to characterize the average attention level of students in this class. The first teaching score of the classroom focus curve is obtained based on the attention resilience index and the preset attention scoring rules.

7. The classroom assessment method based on multimodal data as described in claim 6, characterized in that, The attention resilience index is obtained using the following formula: in, The attention resilience index, This represents the average group attention intensity at the second instant. The average value of the group's attention intensity at the first instant. This is the first total duration. This is the second total duration.

8. A classroom assessment system based on multimodal data, characterized in that, The system includes: a video acquisition module, a content extraction module, and a scoring module; The video acquisition module is used to acquire classroom teaching videos of teachers and classroom status videos of students during the same time period. The content extraction module is used to extract teacher behavior and teaching content from the classroom teaching video, and to extract student behavior from the class status video; the teacher behavior includes the number of times the teacher walks around the podium, the time of walking around, and the number of times the teacher writes on the blackboard; the teaching content includes the content of the blackboard writing, the content of the teaching PPT, and the average speaking speed; the student behavior includes student facial expressions, student head-up rate, and the number of interactions. The scoring module is used to quantitatively evaluate the teacher's behavior, the teaching content, and the student's behavior based on the first preset evaluation rules, and to obtain the evaluation score for this class.

9. A classroom assessment device based on multimodal data, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images