Classroom analysis method and computer device based on computer vision and large language model

Abstract

The present invention relates to the field of classroom structure analysis, and more specifically, to a classroom analysis method and computer device based on computer vision and a large language model, which improves the efficiency and accuracy of classroom analysis. The solution includes: generating multidimensional information, processing user-input teaching videos, lesson plans, and subject information; detecting teacher and student behavior through an SCB model; extracting blackboard images and converting them into structured data; and converting speech content into text; generating prompt words, integrating the multidimensional information including behavior time periods, blackboard time periods, transcription content, lesson plans, and subject information to generate prompt words. The prompt words are input into a large language model to obtain classroom structure content, review the classroom structure content according to pre-examination evaluation criteria, and propose corresponding improvement suggestions based on the review results. The present invention is suitable for classroom structure analysis.

Description

Technical Field

[0001] The present invention relates to the field of classroom structure analysis, and in particular to a classroom analysis method and computer device based on computer vision and a large language model. Background Art

[0002] In traditional teaching models, teachers are the dominant force in the classroom, observing students' expressions, postures, and behaviors to determine whether they understand the content and are interested in learning. Teachers also provide timely feedback and guidance based on student participation and interaction. This method of observation, recording, and feedback helps teachers understand students' learning status and adjust teaching strategies.

[0003] Homework is an important means of assessing students' learning outcomes. In traditional teaching, teachers spend a considerable amount of time and energy grading students' homework and evaluating their learning outcomes. By carefully analyzing students' homework, teachers can identify problems and deficiencies in their learning process, allowing them to provide targeted guidance and instruction.

[0004] Examinations are a crucial way to assess student learning outcomes. In traditional teaching, teachers conduct a comprehensive analysis and summary based on student test scores. They analyze the difficult and error-prone areas of the exam, identifying common student problems and addressing them in subsequent instruction. Furthermore, teachers adjust teaching plans and schedules based on the distribution of student scores to ensure quality instruction.

[0005] With the development of information technology, the application of artificial intelligence in education is becoming increasingly widespread. However, current classroom analysis often relies on manual observation and recording, which is inefficient and susceptible to subjective factors. Computer vision and natural language processing technologies are not fully utilized to achieve automated and accurate analysis of classroom teaching structure. Therefore, a classroom structure analysis method that combines computer vision and large language models is urgently needed to improve the efficiency and accuracy of classroom analysis and assist teachers in optimizing instructional design. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a classroom analysis method and computer device based on computer vision and a large language model, thereby improving the efficiency and accuracy of classroom analysis.

[0007] The present invention adopts the following technical solutions to achieve the above-mentioned objectives. In a first aspect, the present invention provides a classroom analysis method based on computer vision and a large language model, comprising:

[0008] S1, multi-dimensional information generation;

[0009] Users input teaching videos, lesson plans, and subjects;

[0010] The classroom scene data in the teaching video is labeled, and the labeled classroom scene data is used to train the YOLOv7 model. The teaching video is then detected by the trained YOLOv7 model, and the behavior detection information of each frame is stored in a behavior table. The behavior table contains behavior information and the number of occurrences of the behavior. The duration of each behavior is the length of a video. The behaviors include individual behaviors and group behaviors. The number of individual behaviors is greater than or equal to 1, and the number of individual behaviors represents how many students have this behavior. The number of group behaviors is either 0 or 1, where 0 represents that the group does not have this behavior, and 1 represents that the group has this behavior.

[0011] The paraformer model is used to extract the speech content in the teaching video, and the speech transcription model is used to convert the speech content into text content for teachers and students.

[0012] S2, generating prompt words based on multi-dimensional information;

[0013] Integrate the behavior time period, blackboard writing time period, transcription content, lesson plan, and subject information to generate prompt words to guide the large language model, including:

[0014] Based on the content converted by the speech transcription model, the corresponding content in the behavior time period and the blackboard writing time period is filled into the timeline of the content converted by the speech transcription model, forming multiple columns of timeline data, including start time, end time, text content, speaker, behavior, and blackboard writing;

[0015] Generate prompt words for guiding the large language model based on the multiple columns of timeline data, input the prompt words into the large language model, and output the classroom structure content;

[0016] S3. Review the class structure and content based on the pre-screening evaluation criteria;

[0017] The evaluation criteria include:

[0018] The class structure and timeline must be complete, and the entire time period must be greater than 40 minutes;

[0019] The class is divided into multiple teaching sessions, with a total of no less than 8 teaching sessions, and each teaching session lasting no less than 1 minute;

[0020] The teaching process, teaching content and teaching methods should be adjusted appropriately according to the behavior and the content of the blackboard;

[0021] S4. Propose corresponding improvement suggestions based on the review results;

[0022] The improvement suggestions include:

[0023] Select any two teaching links from multiple teaching links, and add time periods for students to write and read in the two selected teaching environments.

[0024] Furthermore, the classroom scene data includes raising hands, reading, writing, discussing, speaking, lowering the head, turning the head, answering, on-stage interaction, blackboard writing, standing, teacher, screen and blackboard data.

[0025] Furthermore, step S1 further includes:

[0026] Draw a student individual behavior curve chart based on the data in the behavior table, draw a student group behavior curve chart based on the data in the behavior table, and observe the time distribution of classroom teaching based on the student individual behavior curve chart and the student group behavior curve chart.

[0027] Furthermore, the behavior information includes: writing on the blackboard, discussing, raising hands, listening, checking, reading aloud, reciting, teaching and writing.

[0028] In a second aspect, the present invention provides a computer device comprising a memory storing program instructions, which, when executed, execute the classroom analysis method based on computer vision and a large language model as described above.

[0029] The beneficial effects of the present invention are:

[0030] The present invention labels classroom scene data in teaching videos, uses the labeled classroom scene data to train a YOLOv7 model, then detects the teaching videos through the trained YOLOv7 model, stores behavior detection information of each frame in a behavior table, wherein the behavior table includes behavior information and the number of occurrences of the behavior, extracts voice content in the teaching videos through a paraformer model, converts the voice content into text content of teachers and students through a voice transcription model, obtains multi-dimensional information such as behavior time period, blackboard writing time period, transcription content, lesson plan and subject, generates prompt words through the multi-dimensional information, inputs the prompt words into a large language model, outputs classroom structure content, and finally reviews, evaluates and improves the classroom structure content according to a pre-examination evaluation standard, thereby improving the efficiency and accuracy of classroom analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of a classroom structure analysis method based on computer vision and a large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] The present invention provides a classroom analysis method based on computer vision and large language model, such as Figure 1 As shown, including:

[0034] S1, multi-dimensional information generation;

[0035] Users input teaching videos, lesson plans, and subjects into the SCB (Student Class Behavior) model. The SCB model is trained based on YOLOv7 and can detect teacher and student behavior in teaching videos and output behavior time period information.

[0036] Specifically, the classroom scene data in the teaching video are annotated, and the classroom scene data include raising hands, reading, writing, discussing, speaking, lowering the head, turning the head, answering, interacting on the stage, writing on the blackboard, standing, teacher, screen, blackboard, etc.

[0037] The labeled classroom scene data is then used to train the YOLOv7 model. The trained YOLOv7 model is then used to detect the teaching video. The behavior detection information for each frame is stored in a behavior table. The behavior table contains behavior information and the number of occurrences of the behavior. The duration of each behavior is the length of a video (in seconds). The present invention divides student or teacher behavior into individual behavior and group behavior. The number of individual behaviors is greater than or equal to 1, representing how many students exhibit this behavior. The number of group behaviors is either 0 or 1, with 0 representing the absence of the group behavior and 1 representing the presence of the group behavior. Group behaviors include listening, reading, and writing. Individual behaviors include raising hands, responding, lecturing, and writing on the blackboard. The output of the classroom video behavior detection sequence is shown in Table 1.

[0038] Table 1 Behavior table

[0039] answer 0 1 1 1 0 1 0 1 2 0 0 0 0 0 ... writing on the blackboard 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... discuss 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... Raise your hand 1 0 0 0 0 0 0 0 0 1 1 1 1 1 ... listen 1 1 1 1 0 0 1 0 0 0 1 1 1 1 ... Conduct an inspection 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... interactive 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... Read aloud 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... recite 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... teaching 1 1 1 0 1 1 1 1 1 1 1 1 1 1 ... Write 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ...

[0040] When a number such as 1, 2, 3, or 4 appears in the current column, it represents the number of times the behavior occurs in the current second.

[0041] This invention can also use the data in Table 1 to visualize behavior across the entire classroom. By plotting individual student behaviors as curves, we can visually observe the fluctuations in student behavior. We also plot multiple horizontal bar charts of group behavior. The sum of all group behavior bar charts equals the class duration, making it easier to observe the time distribution of classroom instruction.

[0042] When the present invention uses the SCB model to detect teaching videos, one of the detections is blackboard writing. The present invention extracts the detection image of the blackboard writing, then inputs the blackboard writing + prompt words into the visual macro model to obtain the structured data of the blackboard writing. Finally, the paraformer model is used to extract the voice content in the video and convert it into text content for teachers and students through the voice transcription model, as shown in Table 2.

[0043] Table 2 Results of speech transcription of classroom teaching videos

[0044]

[0045] S2, generating prompt words based on multi-dimensional information;

[0046] The behavior time period, blackboard writing time period, transcription content, lesson plan, and subject information are integrated to generate prompts for the large language model. First, using the video transcription content as the base timeline, the corresponding content from the behavior and blackboard writing time periods is added to the video transcription timeline, forming a six-column timeline data: start time, end time, text content, speaker, behavior, and blackboard writing. This allows the large language model to capture multi-dimensional information: classroom language information, visual information about student and teacher behavior, and information about the writing on the blackboard.

[0047] Specifically, the prompt words are as follows:

[0048] Analyze the classroom teaching content structure of the teacher according to the following requirements.

[0049] Finally, the output is in JSON format.

[0050] Based on the following classroom video transcription content, behavior time period, blackboard writing time period, and lesson plan content, the subject of the transcribed classroom teaching content is: {{Subject}},

[0051] Based on the classroom teaching content, please output a segmented summary of the class covering all time periods of this class.

[0052] Below is the class video transcription, behavior, and blackboard content: {{Transcript content}}

[0053] Here is the lesson plan for the class video: {{Lesson Plan}}

[0054] Input the above prompt words into the large language model to generate preliminary classroom structure content.

[0055] S3. Review the class structure and content based on the pre-screening evaluation criteria;

[0056] The prompt words are as follows:

[0057] {{Classroom structure and content}}

[0058] This is an analysis and summary of the classroom teaching content structure. Please review this summary strictly in accordance with the following standards and put forward targeted improvement suggestions.

[0059] Examples of evaluation criteria are as follows:

[0060] 1. The timeline of the class structure is complete, and the last time point is greater than 40 minutes, such as "time": "00:39:58-00:43:14", 00:43:14 is greater than 40 minutes, which meets the annotation requirements.

[0061] 2. The class is divided into multiple teaching links, and the total number of teaching links is not less than 8, that is, the id of the last teaching link (id is 0, 1, 2...) is greater than 7.

[0062] 3. The duration of each teaching session is not less than 1 minute. For example, the time: "00:39:58-00:43:14" is 3 minutes and 6 seconds, which is longer than one minute and meets the labeling requirements.

[0063] 4. The time in the classroom structure analysis results may not be very accurate, or the teaching links, teaching content, and teaching methods can be appropriately adjusted based on the behavior in the video and the content on the blackboard.

[0064] S4. Propose corresponding improvement suggestions based on the review results;

[0065] The improvement suggestions include:

[0066] Select any two teaching links from multiple teaching links, and add time periods for students to write and read in the two selected teaching environments.

[0067] Specifically, integrate the improvement suggestions and class structure content into the following prompt words:

[0068] {{Improvement Suggestions}}

[0069] Please improve the content of the following classroom structure based on the above suggestions. If there are no relevant suggestions for improvement, no modification is required.

[0070] {{Classroom structure and content}}

[0071] The final version is presented in JSON format and contains detailed information such as teaching sessions, time, teaching content, and teaching methods. For example: [

[0073] {

[0074] "id":0,

[0075] "Teaching session": "Introduction to new lesson",

[0076] "Time":"00:00:00-00:01:17",

[0077] "Teaching content": ["Show photos of footprints and guide students to guess who the footprints belong to", "Let students touch the cover of a math book to feel the surface of objects", "Ask students how one side of an orange is different from other sides"],

[0078] "Teaching method": "Visual demonstration, question and answer"

[0079] }, ... ]

[0082] The present invention realizes a comprehensive and accurate analysis of classroom teaching through the above four stages, which can assist teachers in optimizing teaching design and improving teaching quality.

[0083] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A classroom analysis method based on computer vision and large language models, characterized by: include: S1, multi-dimensional information generation; Users input teaching videos, lesson plans, and subjects; The classroom scene data in the teaching video is labeled, and the labeled classroom scene data is used to train the YOLOv7 model. The teaching video is then detected by the trained YOLOv7 model, and the behavior detection information of each frame is stored in a behavior table. The behavior table contains behavior information and the number of occurrences of the behavior. The duration of each behavior is the length of a video. The behaviors include individual behaviors and group behaviors. The number of individual behaviors is greater than or equal to 1, and the number of individual behaviors represents how many students have this behavior. The number of group behaviors is either 0 or 1, where 0 represents that the group does not have this behavior, and 1 represents that the group has this behavior. Extract the blackboard writing images from the teaching video and input them into the visual model to obtain the structured data of the blackboard writing; The paraformer model is used to extract the speech content in the teaching video, and the speech transcription model is used to convert the speech content into text content for teachers and students. Behavior time period, blackboard writing time period, transcription content, lesson plan and subject information are multi-dimensional information; S2, generating prompt words based on multi-dimensional information; Integrate the behavior time period, blackboard writing time period, transcription content, lesson plan, and subject information to generate prompt words to guide the large language model, including: Based on the content converted by the speech transcription model, the corresponding content in the behavior time period and the blackboard writing time period is filled into the timeline of the content converted by the speech transcription model, forming multiple columns of timeline data, including start time, end time, text content, speaker, behavior, and blackboard writing; Generate prompt words for guiding the large language model based on the multiple columns of timeline data, input the prompt words into the large language model, and output the classroom structure content; S3. Review the class structure and content based on the pre-screening evaluation criteria; The evaluation criteria include: The class structure and timeline must be complete, and the entire time period must be greater than 40 minutes; The class is divided into multiple teaching sessions, with a total of no less than 8 teaching sessions, and each teaching session lasting no less than 1 minute; The teaching process, teaching content and teaching methods should be adjusted appropriately according to the behavior and the content of the blackboard; S4. Propose corresponding improvement suggestions based on the review results; The improvement suggestions include: Select any two teaching links from multiple teaching links, and add time periods for students to write and read in the two selected teaching environments.

2. The classroom analysis method based on computer vision and large language model according to claim 1 is characterized in that: The classroom scene data includes raising hands, reading, writing, discussing, speaking, lowering the head, turning the head, answering, on-stage interaction, blackboard writing, standing, teacher, screen and blackboard data.

3. The classroom analysis method based on computer vision and large language model according to claim 1 is characterized in that: Step S1 further includes: Draw a student individual behavior curve chart based on the data in the behavior table, draw a student group behavior curve chart based on the data in the behavior table, and observe the time distribution of classroom teaching based on the student individual behavior curve chart and the student group behavior curve chart.

4. The classroom analysis method based on computer vision and large language model according to claim 1 is characterized in that: The behavior information includes: writing on the blackboard, discussing, raising hands, listening, checking, reading aloud, reciting, teaching, and writing.

5. A computer device comprising a memory storing program instructions, characterized in that: When the program instructions are executed, the classroom analysis method based on computer vision and large language model as described in any one of claims 1 to 4 is executed.

Images