A method and system for automatically detecting classroom teaching types based on multimodal data

Abstract

The present invention relates to the fields of computer vision and natural language processing, and provides a method and system for automatically detecting classroom teaching types based on multimodal data. The method comprises: (1) constructing a classroom teaching type detection indicator system; (2) establishing a classroom teaching type detection indicator calculation method; and (3) classroom teaching type detection and judgment. The present invention provides an automated method for detecting and judging classroom teaching types, achieving intelligent recognition and detection of classroom type-related features.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and natural language processing, and in particular to a method and system for automatically detecting classroom teaching types based on multimodal data. Background Art

[0002] As a major aspect of teaching performance, classroom teaching type can be studied to conduct stratified and diversified personalized classroom teaching quality analysis and promote the improvement of classroom teaching quality. The detection of classroom teaching type based on multimodal data processing technology fully utilizes the advantages of deep learning in the fields of computer vision and natural language processing, which is conducive to a comprehensive and accurate understanding of classroom teaching types and provides support for promoting effective teaching and building high-quality classrooms. The following difficulties exist in the automatic detection of classroom teaching types based on multimodal technology: (1) Current classroom video resources are limited by shooting angles, student distribution interference, and low video quality. It is difficult for machines to automatically and accurately identify teaching actions and audio data, and it is difficult to accurately extract and identify classroom details; (2) There is a lack of feature recognition and indicator construction for classroom teaching types, which makes it difficult to achieve comprehensive and systematic classroom teaching type detection and judgment; (3) There is a lack of standardized processes and methods for classroom teaching type detection based on multimodal data, which makes it difficult to achieve large-scale, automated, and objective classroom teaching type detection and classification. Summary of the Invention

[0003] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and system for automatic detection of classroom teaching types based on multimodal data. By automatically identifying and judging the teacher and student movements, positions and classroom communication data in classroom recording video resources, the characteristics of classroom teaching types are extracted, and intelligent detection and accurate judgment of classroom teaching types are realized, providing support for a comprehensive and accurate understanding of classroom teaching types, promoting efficient teaching of teachers, and building high-quality classrooms.

[0004] The objectives of the present invention are achieved through the following technical measures.

[0005] A method for automatically detecting classroom teaching types based on multimodal data includes the following steps:

[0006] (1) Construction of classroom teaching type detection indicator system. According to the characteristics of different classroom teaching types, classroom teaching type detection indicators are determined and the detection points of each detection indicator are established;

[0007] (2) Establishment of the method for calculating the classroom teaching type detection indicators. Based on the data modality processing technology involved in the classroom teaching type detection indicators, image processing models and language processing models are established respectively to determine the calculation methods for each detection indicator;

[0008] (3) Classroom teaching type detection and judgment: read the classroom video resources to be detected, use the classroom teaching type detection index measurement method to obtain the detection index measurement results, judge the classroom teaching type, and display the detection index measurement results and classroom teaching type judgment results in a visual way.

[0009] The present invention also provides a system for automatically detecting classroom teaching types based on multimodal data, which is used to implement the above-mentioned method for automatically detecting classroom teaching types based on multimodal data, and includes the following modules:

[0010] The classroom teaching type detection indicator module establishes various indicators for classroom teaching type detection based on the characteristics of classroom teaching types, and confirms the detection points of each indicator.

[0011] The language processing model module builds and trains a language processing model for measuring classroom teaching type detection indicators.

[0012] Image processing model module, builds and trains image processing models for measuring classroom teaching type detection indicators.

[0013] The classroom teaching type detection indicator calculation method module constructs the calculation method for various detection indicators.

[0014] The classroom teaching type detection and judgment module uses the trained classroom teaching type detection index calculation method to process the classroom recording video resources to be detected, obtains the classroom teaching type detection index calculation results, and judges the classroom teaching type based on the calculation results.

[0015] The beneficial effects of the present invention are:

[0016] Applying computer image recognition technology and natural language processing technology, the image and voice information of teachers and students in classroom recordings are captured for detection and processing, the characteristics of classroom teaching types are identified, classroom teaching types are intelligently classified, and visual results are displayed. It supports real-time detection and automatic feedback of classroom teaching types, which is conducive to a comprehensive and accurate understanding of classroom teaching types, and promotes the diversification of teachers' teaching activities and the construction of high-quality classrooms. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the automatic detection method for classroom teaching types according to an embodiment of the present invention.

[0018] Figure 2 Schematic diagram of the network structure of the YOLO-ST model in an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of the network structure of the speech-to-text model in an embodiment of the present invention.

[0020] Figure 4 It is a schematic diagram of the calculation process of the "student distribution" detection indicator in an embodiment of the present invention.

[0021] Figure 5 It is a schematic diagram of the "podium role" detection index calculation process of an embodiment of the present invention.

[0022] Figure 6 It is a schematic diagram of the flow chart of the “teacher’s lecture” detection indicator calculation process according to an embodiment of the present invention.

[0023] Figure 7 It is a schematic diagram of the calculation process of the "key and difficult point association" detection index in an embodiment of the present invention.

[0024] Figure 8 It is a schematic diagram of the flow chart of the “student questioning” detection index calculation process according to an embodiment of the present invention.

[0025] Figure 9 It is a schematic diagram of the flow chart of the “classroom interaction” detection indicator calculation process according to an embodiment of the present invention.

[0026] Figure 10 It is a visualization diagram of the classroom teaching type detection and judgment results of an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0028] like Figure 1 As shown, an embodiment of the present invention provides a method for automatically detecting classroom teaching types based on multimodal data, comprising the following steps:

[0029] (1) Construction of classroom teaching type detection indicator system. According to the characteristics of different classroom teaching types, classroom teaching type detection indicators are determined, and the detection points of each detection indicator are established.

[0030] (1-1) Determine the characteristics of classroom teaching types. Classroom teaching types are divided into five categories: knowledge teaching classes, report presentation classes, question answering classes, communication and discussion classes, and other classes. The main characteristics of each classroom teaching type are as follows:

[0031] Characteristics of knowledge teaching classes: mainly teachers imparting knowledge.

[0032] Characteristics of report and demonstration classes: mainly students reporting their learning outcomes.

[0033] The characteristics of question-and-answer classes: teachers mainly answer students' questions.

[0034] Characteristics of communication and discussion classes: mainly based on students’ mutual communication and collaboration.

[0035] Characteristics of other courses: do not meet the above characteristics.

[0036] (1-2) Determine relevant indicators for course type testing. Determine the testing indicators for classroom teaching types. Based on the characteristics of each classroom teaching type mentioned above, determine the testing indicators and key points for each indicator.

[0037] (1-2-1) Determine assessment indicators. Establish assessment indicators based on the teacher and student positions and activities during classroom instruction. Specifically, these indicators include: student distribution, podium roles, teacher presentation, correlation between key and difficult points, student questions, and classroom interaction. The specific meanings of each evaluation indicator are shown in Table 1.

[0038] Table 1 Classroom teaching type detection indicators

[0039]

[0040] (1-2-2) Determine the key points for testing each indicator. Based on the meaning and content of each test indicator, determine the key points for testing each indicator, as follows:

[0041] Key points for testing the "student distribution" indicator: Based on the distance between students, determine whether the student distribution is grouped or not.

[0042] Key points for testing the “podium role” indicator: Based on whether the characters around the podium are teachers or students, calculate the length of time the teacher and students spend at the podium.

[0043] Key points for testing the "Teacher Lecture" indicator: Calculate the teacher's lecture time based on the teacher's audio content and course syllabus documents, and determine whether the teacher's lecture content covers the key points and difficulties of the class.

[0044] Key points for testing the "correlation between key points and difficult points" indicator: Based on the teacher's audio content, the student's audio content and the course syllabus document, calculate the correlation between the teacher-student question and answer content and the key points and difficult points in the class.

[0045] Key points for testing the "Student Questioning" indicator: Based on the students' questioning actions in class, calculate the number of students asking questions in class.

[0046] Key points for testing the "classroom interaction" indicator: Calculate the duration of teacher-student interaction and student-student interaction based on the teacher's audio content and the student's audio content.

[0047] (2) Establish a method for calculating classroom teaching type detection indicators. Based on the data modality processing technology involved in classroom teaching type detection indicators, establish image processing models and language processing models respectively, and determine the calculation methods for each detection indicator.

[0048] (2-1) Classification of classroom teaching type detection indicators. Based on the data modality processing technology involved in the key points of the indicator detection, it is divided into two types: image processing and language processing. The specific indicators of image processing and language processing are shown in Table 2.

[0049] Table 2 Classification of classroom teaching type detection indicators

[0050]

[0051] (2-2) Constructing a detection index processing model. For the two types of detection indicators, image processing and language processing, an image processing model and a language processing model were constructed.

[0052] (2-2-1) Image processing model. There are two types of image processing models: one is the existing general pre-trained model YOLO-Face; the other is a dedicated model. The network structure of the dedicated model is to add the Swin-Transformer module to the network structure of the YOLOV5 model and train it to produce the YOLO-ST model. The YOLO-ST network structure is as follows: Figure 2 As shown, the workflow is as follows:

[0053] A. Input: Classroom teaching type training samples are adaptively scaled to 608*608*3 data and output to the BackBone-Swin network.

[0054] B. The data is sliced and fused in the Focus module in BackBone-Swin. The CBL module and CSP1_5 module then perform multiple feature upsampling and downsampling. The SPP module performs maximum pooling, followed by feature fusion and output to the Swin-Transformer module. The data is then exported to the Neck network.

[0055] C. The data is aggregated through different tensor concatenation layers in the Neck network to generate the image to be predicted and pass it to the output end.

[0056] D. The output end obtains the feature data from the Neck network, then uses the feature data to generate prediction results and calculate the loss value of the prediction results.

[0057] (2-2-2) Language processing model. The language processing model is a speech-to-text model generated by training the Seq2SeqAttention network structure, which extracts the teacher-student speech content from the teacher-student audio. The network structure of the speech-to-text model is as follows: Figure 3 As shown, the workflow is as follows:

[0058] A. The text data used for training is imported into the AttenDecoderRNN module. The specific process is as follows: First, it is input into the embedding layer for feature extraction to obtain 1*1*256 feature data. This feature data is then input into the dropout layer to hide some features to obtain word1. The word1 and the hidden layer data are then concatenated into tensors. The concatenated result is processed by the adaptive attention layer and then by the softmax layer to obtain the weight value of the attention information.

[0059] B. The speech data used for training is imported into the EncoderRNN module. The specific process is as follows: the speech data is first input into the embedding layer for feature extraction to obtain 1*1*256 feature data. It is then further extracted through the LSTM layer and the results are output to the AttenDecoderRNN.

[0060] C. Perform matrix multiplication on the weight value and the data output by EncoderRNN to obtain the 1*1*256 feature data word2; then concatenate word1 and word2 into tensors, and then pass the concatenated result through the attention combination layer, and then through the relu function for activation processing, and then pass the processed result through LSTM to further extract features, and finally give it to softmax for result prediction.

[0061] (2-3) Determine the method for calculating the indicators for classroom teaching type detection. Extract the teacher-student audio and course syllabus files from the classroom video resources to be tested, convert the classroom video resources to be tested into a set of images P with uniform specifications, and record the corresponding time of each image. Then, establish the calculation method for each indicator separately, as follows:

[0062] The method for calculating the “student distribution” detection indicator is as follows: for each picture in the picture set P, the YOLO-ST model is used to identify and detect the coordinate position of each student; then, the system clustering algorithm is used to calculate the number of student clusters based on the coordinate position of each student; finally, for the number of student clusters detected for each picture in the picture set P, the mode is selected as the detection result N distribution , the specific flow chart is as follows Figure 4 shown.

[0063] The method for calculating the “podium role” detection indicator is as follows: for each image in the image set P, use YOLO-ST to detect the podium and obtain the coordinate position of the podium on the image; then, use the YOLO-Face model to determine whether the person around the podium is a teacher role or a student role, and obtain the podium role corresponding to each moment; finally, calculate the length of time the teacher role and the student role are on the podium respectively. The specific flow chart is as follows: Figure 5 shown.

[0064] The method for calculating the “teacher lecture” detection index is as follows: the teacher’s speech content and the teacher’s audio duration T are extracted from the teacher-student audio using the speech-to-text model. teach Extract key points from the course syllabus; use TF-IDF to calculate whether the teacher's speech content contains key points. The specific flow chart is as follows: Figure 6 shown.

[0065] Method for calculating the “key and difficult point correlation” detection index: Using the speech-to-text model, extract the teacher-student speech content from the teacher-student audio, filter the teacher-student Q&A related speech, and calculate the total duration T of the teacher-student Q&A audio. Q-S ; Extract the key points of the class from the course syllabus file; then use TF-IDF to calculate the similarity between the teacher-student Q&A related discourse and the key points of the class, and get the similarity R com , the specific flow chart is as follows Figure 7 shown.

[0066] The method for calculating the “student questioning” detection indicator is as follows: for each picture in the picture set P, use the YOLO-ST model to detect the student’s hand-raising action; for each picture in the picture set P, the set of student hand-raising coordinates is obtained, and the number of students asking questions N is calculated. ask , the specific flow chart is as follows Figure 8 shown.

[0067] Classroom interaction: Method for calculating the “classroom interaction” detection index: Use the speech-to-text model to extract the teacher-student speech content and calculate the teacher-student interaction time T T-S Interaction time with students T S-S , the specific flow chart is as follows Figure 9 shown.

[0068] (3) Classroom teaching type detection and judgment. Read the classroom video resources to be detected, use the classroom teaching type detection indicator measurement method to obtain the detection indicator measurement results, judge the classroom teaching type, and display the detection indicator measurement results and classroom teaching type judgment results in a visual way.

[0069] (3-1) Calculate the classroom teaching type detection index. Call the classroom teaching type detection index calculation method, analyze the classroom recording video resources to be tested, and obtain the classroom teaching type detection index calculation results.

[0070] (3-2) Determine the type of classroom teaching. Based on the results of the classroom teaching type detection indicators, determine the type of classroom teaching, as follows:

[0071] A. Judgment basis for “knowledge teaching class”: In the measurement results of the “lecture role” indicator, the lecturer’s teaching time is greater than 80% of the total class time; in the measurement results of the “teacher teaching” indicator, the teacher’s teaching time is greater than 80% of the total class time. teach More than 80% of the total class time, the teacher's lecture content includes the key and difficult points of the class.

[0072] B. Judgment basis for “Report and Demonstration Class”: In the measurement results of the “Lecture Role” indicator, the student duration is greater than 50% of the total course duration; in the measurement results of the “Teacher Lecture” indicator, the teacher lecture duration is greater than 50% of the total course duration. teach Less than 60% of the total class duration.

[0073] C. Judgment basis for “Question-answering and doubt-solving class”: In the measurement results of the “classroom interaction” indicator, the teacher-student interaction time T T-S More than 50% of the total class time; in the “student questions” indicator measurement results, the number of students asking questions N ask Greater than 3; in the calculation results of the “key and difficult point correlation” indicator, the key and difficult point similarity R com Greater than 50%.

[0074] D. Judgment basis for “communication and discussion class”: In the calculation results of the “student distribution” indicator, N distribution Greater than 3; in the “classroom interaction” indicator measurement results, the student-student interaction time T S-S More than 50% of the total class time.

[0075] E. Judgment basis for “other courses”: the indicator measurement results do not meet any of the above requirements.

[0076] (3-3) Visualization. Combining the results of classroom teaching type detection index calculation and classroom teaching type judgment, a line chart and classroom video screenshots are used for visualization. The diagram of the visualization results is as follows: Figure 10 shown.

[0077] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0078] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatic detection of classroom teaching types based on multimodal data, characterized by The method comprises the following steps: (1) Construction of classroom teaching type detection indicator system. According to the characteristics of different classroom teaching types, classroom teaching type detection indicators are determined, and the detection points of each detection indicator are established; according to the meaning and content of each detection indicator, the detection points of each detection indicator are determined, as follows: Key points for testing the "student distribution" indicator: Based on the distance between students, determine whether the student distribution is clustered; Key points for testing the "podium role" indicator: Based on whether the characters around the podium are teachers or students, calculate the length of time the teacher and students spend at the podium; Key points for testing the "Teacher Lecture" indicator: Calculate the teacher's lecture time based on the teacher's audio content and the course syllabus, and determine whether the teacher's lecture content covers the key points of the class; Key points for testing the "Key Points Correlation" indicator: Calculate the correlation between teacher-student Q&A content and key points and difficulties in class based on the teacher's audio content, student audio content, and the course syllabus. Key points for testing the "Student Questioning" indicator: Calculate the number of students asking questions in class based on their questioning behavior; Key points for testing the "classroom interaction" indicator: Calculate the duration of teacher-student and student-student interaction based on the teacher's and student's audio content; (2) Establishment of the method for calculating the classroom teaching type detection indicators. Based on the data modality processing technology involved in the classroom teaching type detection indicators, image processing models and language processing models are established respectively to determine the calculation methods for each detection indicator. The details are as follows: (2-1) Classification of classroom teaching type detection indicators: Based on the data modality processing technology involved in the key points of indicator detection, it is divided into two types: image processing and language processing; (2-2) Constructing a detection index processing model. For the two types of detection indicators, image processing and language processing, an image processing model and a language processing model were constructed; (2-2-1) Image processing model. There are two types of image processing models: one is the existing general pre-trained model YOLO-Face; the other is a dedicated model. The network structure of the dedicated model is to add the Swin-Transformer module to the network structure of the YOLOV5 model, and train it to produce the YOLO-ST model. (2-2-2) Language processing model: The language processing model is a speech-to-text model generated by training the Seq2SeqAttention network structure, which extracts the teacher-student speech content from the teacher-student audio; (2-3) Determine the method for measuring the indicators of classroom teaching type detection. To determine the method for measuring the indicators of classroom teaching type detection, extract the teacher-student audio and course syllabus files from the classroom video resources to be detected, convert the classroom video resources to be detected into a set of pictures of uniform specifications P, record the corresponding time of each picture, and then establish the measurement method for each indicator respectively; (3) Classroom teaching type detection and judgment: read the classroom video resources to be detected, use the classroom teaching type detection index measurement method to obtain the detection index measurement results, judge the classroom teaching type, and display the detection index measurement results and classroom teaching type judgment results in a visual way.

2. The automatic detection method of classroom teaching type based on multimodal data according to claim 1 is characterized in that The specific process of constructing the classroom teaching type detection indicator system in step (1) is as follows: (1-1) Determine the characteristics of classroom teaching types. Classroom teaching types are divided into five categories: knowledge teaching classes, report presentation classes, question answering classes, communication and discussion classes, and other classes. The characteristics of each classroom teaching type are as follows: Characteristics of knowledge-teaching courses: including teachers teaching knowledge; Characteristics of presentation lessons: including students reporting on their learning outcomes; Characteristics of question-answering classes: including teachers answering students’ questions; Characteristics of communication and discussion classes: including students communicating and collaborating with each other; Characteristics of other classes: do not meet the above characteristics; (1-2) Determine the testing indicators for classroom teaching types. Based on the characteristics of each classroom teaching type mentioned above, determine the testing indicators and key points for each indicator; (1-2-1) Determine the testing indicators. Establish testing indicators from two perspectives: the positions of teachers and students and their activities and behaviors during classroom teaching. Specifically, these indicators include: student distribution, podium roles, teacher lectures, the relevance of key points and difficulties, student questions, and classroom interaction. The specific meanings and contents of each indicator are as follows: "Student Distribution" indicator: used to detect the distribution of students in the classroom; "Pedestrian role" indicator: used to detect the specific identity of people around the podium; "Teacher Lecture" indicator: used to measure the duration of teacher lectures, the content of teacher lectures, and the closeness of key points and difficulties in class; "Key and difficult points correlation" indicator: used to measure the degree of closeness between the teacher-student classroom interaction content and the key and difficult points of the class; "Student Questions" indicator: used to detect the number of students asking questions in class; "Classroom interaction" indicator: used to measure the duration of teacher-student interaction and student-student interaction; (1-2-2) Determine the key points for testing each indicator, and determine the key points for testing each indicator based on its meaning and content.

3. The automatic detection method of classroom teaching type based on multimodal data according to claim 1 is characterized in that The specific process of establishing the measurement method for each indicator in step (2-3) is as follows: The "student distribution" detection indicator calculation method is as follows: for each picture in the picture set P, the YOLO-ST model is used to identify and detect the coordinate position of each student; then, the system clustering algorithm is used to calculate the number of student clusters based on the coordinate position of each student; finally, for the number of student clusters detected for each picture in the picture set P, the mode is selected as the detection result N distribution ; Method for calculating the "podium character" detection metric: For each image in the image set P, use YOLO-ST to detect the podium and obtain its coordinates on the image. Then, use the YOLO-Face model to determine whether the person around the podium is a teacher or a student, and obtain the corresponding podium character at each moment. Finally, calculate the duration of the teacher and student characters on the podium respectively. The method for calculating the "teacher lecture" detection index is to use the speech-to-text model to extract the teacher's speech content and the teacher's audio duration T from the teacher-student audio. teach Extract key points from the course syllabus and use TF-IDF to calculate whether the teacher's speech content contains key points. Method for calculating the "key and difficult point correlation" detection indicator: Using the speech-to-text model, extract the teacher-student speech content from the teacher-student audio, filter the teacher-student Q&A related speech, and calculate the total duration T of the teacher-student Q&A audio. Q-S ; Extract the key points of the class from the course syllabus file; then use TF-IDF to calculate the similarity between the teacher-student Q&A discourse and the key points of the class, and get the similarity R com ; The method for calculating the "student questioning" detection indicator is as follows: for each picture in the picture set P, use the YOLO-ST model to detect the student's hand-raising action; for each picture in the picture set P, the set of student hand-raising coordinates detected is used to calculate the number of students asking questions N ask ; Method for calculating the "classroom interaction" detection index: Use the speech-to-text model to extract the teacher-student discourse content and calculate the teacher-student interaction time T T-S Interaction time with students T S-S .

4. The automatic detection method of classroom teaching type based on multimodal data according to claim 1 is characterized in that The specific process of classroom teaching type detection and judgment in step (3) is as follows: (3-1) Calculating classroom teaching type detection indicators, calling classroom teaching type detection indicator calculation method, analyzing classroom recording video resources to be detected, and obtaining classroom teaching type detection indicator calculation results; (3-2) Determine the type of classroom teaching. According to the results of the classroom teaching type detection indicators, determine the type of classroom teaching as follows: A. Judgment basis for "knowledge teaching class": In the measurement results of the "lecture role" indicator, the lecturer's teaching time is greater than 80% of the total class time; in the measurement results of the "teacher lecture" indicator, the teacher's teaching time is greater than 80% of the total class time. teach More than 80% of the total class time, and the teacher's lecture content includes the key points and difficult points of the class; B. Judgment basis for "Report and Demonstration Class": In the measurement results of the "Lecture Role" indicator, the student duration is greater than 50% of the total course duration; in the measurement results of the "Teacher Lecture" indicator, the teacher lecture duration is greater than 50% of the total course duration. teach Less than 60% of the total class time; C. Judgment basis for "Question-answering and doubt-solving class": In the measurement results of the "classroom interaction" indicator, the teacher-student interaction time T T-S More than 50% of the total class time; in the "student questions" indicator calculation results, the number of students asking questions N ask Greater than 3; in the calculation results of the "key and difficult point correlation" indicator, the key and difficult point similarity R com greater than 50%; D. Judgment basis for "communication and discussion class": In the calculation results of the "student distribution" indicator, N distribution Greater than 3; in the "classroom interaction" indicator measurement results, the student-student interaction time T S-S More than 50% of the total class time; E. "Other courses" judgment basis: the indicator measurement results do not meet any of the above requirements; (3-3) Visual display: combining the results of classroom teaching type detection index measurement and classroom teaching type judgment results, a comprehensive use of line charts, bar charts, and classroom video screenshots for visual drawing.

5. An automatic detection system for classroom teaching types based on multimodal data, characterized by include: The classroom teaching type detection indicator module establishes various indicators for classroom teaching type detection based on the characteristics of classroom teaching types and confirms the detection points of each indicator; according to the meaning and content of each detection indicator, the detection points of each detection indicator are determined as follows: Key points for testing the "student distribution" indicator: Based on the distance between students, determine whether the student distribution is clustered; Key points for testing the "podium role" indicator: Based on whether the characters around the podium are teachers or students, calculate the length of time the teacher and students spend at the podium; Key points for testing the "Teacher Lecture" indicator: Calculate the teacher's lecture time based on the teacher's audio content and the course syllabus, and determine whether the teacher's lecture content covers the key points of the class; Key points for testing the "Key Points Correlation" indicator: Calculate the correlation between teacher-student Q&A content and key points and difficulties in class based on the teacher's audio content, student audio content, and the course syllabus. Key points for testing the "Student Questioning" indicator: Calculate the number of students asking questions in class based on their questioning behavior; Key points for testing the "classroom interaction" indicator: Calculate the duration of teacher-student and student-student interaction based on the teacher's and student's audio content; The language processing model module builds and trains a language processing model for measuring classroom teaching type detection indicators. The language processing model is a speech-to-text model trained based on the Seq2SeqAttention network structure, which extracts the teacher-student speech content from the audio. The image processing model module builds and trains image processing models for measuring classroom teaching type detection indicators. There are two types of image processing models: one is the existing general pre-trained model YOLO-Face; the other is a dedicated model. The network structure of the dedicated model is based on the YOLOV5 model network structure with the Swin-Transformer module added to train the YOLO-ST model. The classroom teaching type detection indicator calculation method module constructs a calculation method for measuring various detection indicators; extracts the teacher and student audio and course syllabus files from the classroom recording video resources to be tested, converts the classroom recording video resources to be tested into a set of pictures of uniform specifications P, records the corresponding time of each picture, and then establishes a calculation method for each indicator respectively; The classroom teaching type detection and judgment module uses the trained classroom teaching type detection index calculation method to process the classroom recording video resources to be detected, obtains the classroom teaching type detection index calculation results, and judges the classroom teaching type based on the calculation results.

Images