Teacher classroom teaching target teaching feature detection method and system based on multi-modal data

Abstract

The invention relates to the field of classroom teaching monitoring, and provides a teacher classroom teaching target teaching feature detection method and system based on multi-modal data. The method comprises the following steps: (1) determining classroom teaching target detection content; (2) constructing a classroom teaching target content detection standard library; (3) establishing a teacher classroom teaching target teaching feature detection method; and (4) detecting classroom teaching target teaching features of the teacher. According to the multi-modal data-based teacher classroom teaching target teaching feature detection method and system, teaching materials, student information and classroom communication data in classroom teaching resources are automatically extracted and judged, and the teacher classroom teaching target features are extracted, so that intelligent identification and accurate detection of the teacher classroom teaching target are realized, and the teaching efficiency is improved. And support is provided for comprehensively and accurately knowing classroom teaching target teaching features of teachers, promoting efficient teaching of teachers, efficient learning of students and building high-quality classrooms.

Description

Technical Field

[0001] The present invention relates to the field of classroom teaching monitoring, and in particular to a method and system for detecting teaching characteristics of teachers' classroom teaching objectives based on multimodal data. Background Art

[0002] Teaching objectives constitute the core elements of teachers' teaching activities, provide a fundamental basis for the orderly development of classroom education activities, and are the basic basis for classroom teaching. The detection of teaching characteristics of teachers' classroom teaching objectives based on multimodal data processing technology fully utilizes the advantages of deep learning in the fields of natural language processing and computer image processing, which is conducive to a comprehensive and accurate understanding of the teaching characteristics of teachers' classroom teaching objectives, and provides support for promoting effective teaching of teachers, efficient learning of students, and building high-quality classrooms. At present, there are the following difficulties in the detection of teaching characteristics of teachers' classroom teaching objectives based on multimodal data: (1) The current classroom video resources are limited by the shooting angle, student distribution interference, and low video quality. It is difficult for the machine to automatically and accurately identify student movements and audio data, and it is difficult to accurately extract and identify classroom teaching details; (2) There is a lack of content for detecting teaching characteristics of teachers' classroom teaching objectives, which makes it difficult to realize the automatic extraction of comprehensive and systematic characteristics of teachers' classroom teaching objectives; (3) There is a lack of standardized processes and methods for detecting teaching characteristics of teachers' classroom teaching objectives based on multimodal data, which makes it difficult to realize large-scale, automated, and objective automatic detection of teaching characteristics of teachers' classroom teaching objectives. Summary of the invention

[0003] In response to the above defects or improvement needs of the prior art, the present invention provides a method and system for detecting the teaching characteristics of teachers' classroom teaching objectives based on multimodal data. By automatically extracting and judging teaching materials, student information and classroom communication data in classroom teaching resources, the characteristics of teachers' classroom teaching objectives are extracted, and intelligent identification and precise detection of teachers' classroom teaching objectives are achieved. This provides support for comprehensively and accurately understanding the teaching characteristics of teachers' classroom teaching objectives, promoting efficient teaching of teachers, efficient learning of students, and building high-quality classrooms.

[0004] The purpose of the present invention is achieved through the following technical measures.

[0005] The present invention provides a method for detecting characteristics of classroom teaching objectives of teachers based on multimodal data, and the method comprises the following steps:

[0006] (1) Determine the content of classroom teaching goal testing. According to the process and characteristics of course teaching objectives, establish a teacher classroom teaching goal indicator system and determine the specific content of teacher classroom teaching goal teaching characteristics testing.

[0007] (2) Construct a standard library for detecting classroom teaching objectives and content. Construct a standard library for detecting the teaching characteristics of teachers’ classroom teaching objectives and use the content of the standard library as the benchmark for detecting the teaching characteristics of classroom teaching objectives.

[0008] (3) Establish a method for detecting the characteristics of teachers’ classroom teaching objectives. Based on the teacher classroom teaching objective indicator system and the classroom teaching objective content detection standard library, establish a classroom teaching objective feature recognition algorithm and an indicator detection algorithm.

[0009] (4) Detect the teaching characteristics of teachers’ classroom teaching objectives. Read the teaching resources of the teachers to be tested, apply the detection method of teachers’ classroom teaching objectives based on the classroom teaching objectives detection content and the classroom teaching objectives content detection standard library, calculate the relevant data of each indicator, form the comprehensive detection results of teachers’ classroom teaching objectives teaching characteristics, and feedback the detection results to the teachers.

[0010] At the same time, the present invention also provides a teacher classroom teaching goal teaching feature detection system based on multimodal data, including the following modules:

[0011] The indicator detection content module confirms the specific detection content of each indicator and the model to be used.

[0012] The classroom teaching objective content detection standard library module uses a language feature detection model to extract the classroom teaching objective content in the teaching syllabus to form a classroom teaching objective content detection standard library.

[0013] The multimodal data processing module builds a teacher classroom teaching objective teaching feature detection model trained for multimodal data.

[0014] The speaker log confirmation module processes the voice data, gives each person's audio segment, and outputs the corresponding time information.

[0015] The model training module is based on the model network of multimodal data processing, and is used to train the feature detection model for teachers to teach classroom teaching objectives.

[0016] The teacher's classroom teaching objectives and teaching characteristics detection module processes the detection results of various indicators and conducts a comprehensive detection of the teacher's classroom teaching objectives and teaching characteristics.

[0017] The beneficial effects of the present invention are:

[0018] Applying computer image processing technology and natural speech processing technology, the image and voice information of teachers and students in classroom recordings are captured for detection and processing, the characteristics of classroom teaching objectives are identified, classroom teaching objectives are intelligently classified, and the teaching characteristics of teachers' classroom teaching objectives are detected. At the same time, the detection results are fed back in the form of graphics and text, realizing real-time detection and automatic feedback of the teaching characteristics of teachers' classroom teaching objectives, which is conducive to a comprehensive and accurate understanding of the teaching characteristics of teachers' classroom teaching objectives, and promoting teachers' teaching activities and high-quality classroom construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of teacher classroom teaching objective teaching feature detection in an embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of a speaker log confirmation module according to an embodiment of the present invention.

[0021] Figure 3 Schematic diagram of the Meta-Goal-Transformer model of an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the structure of a multimodal data serializer for classroom teaching objective teaching features of an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of graphic feedback of the teaching characteristics of the teacher's classroom teaching objectives in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting teacher classroom teaching objectives and teaching features based on multimodal data, comprising the following steps:

[0026] (1) Determine the content of classroom teaching goal testing. According to the process and characteristics of course teaching objectives, establish a teacher classroom teaching goal indicator system and determine the specific content of teacher classroom teaching goal teaching characteristics testing.

[0027] The specific steps include:

[0028] (1-1) Classroom teaching goal detection content indicators. From the perspective of the teacher's teaching objectives, it includes whether the teacher's teaching enables students to acquire basic knowledge or basic abilities; whether the teacher's teaching includes the indispensable process and methods for solving problems; whether the teacher conveys positive emotional values ​​to students during the teaching process. Based on the above characteristics and required goals, it is confirmed that the teacher's goals detected by this system mainly include: knowledge and skill goals, process and method goals, and emotional attitude and value goals. The details are shown in Table 1:

[0029] Table 1 Classification of classroom teaching objectives

[0030]

[0031] (1-2) Determine the content of the indicator test. According to the meaning of each indicator, determine the relevant test content, as shown in Table 2.

[0032] Table 2 Classroom teaching goal detection content indicators

[0033]

[0034] The test content of the "knowledge and skill goals" indicator: teaching syllabus, teacher classroom teaching video recording. Extract the teacher's speech content, teacher's position and teacher's teaching method in classroom teaching; calculate the degree of match between the teacher's speech content and the student knowledge learning and skill learning goals proposed in the teaching syllabus (the basic concepts, basic principles and basic methods that students should master); determine the teacher's position, teacher's teaching method, and current time period when teaching the content related to this indicator;

[0035] The test content of the "process and method objectives" indicator: teaching syllabus, teacher classroom teaching video recording. Extract the teacher's speech content and the teacher's position in the classroom teaching; calculate the degree of match between the teacher's speech content and the student learning process and method objectives proposed in the teaching syllabus (the way of thinking, learning methods, learning strategies and learning skills that students should master); calculate the teacher's position, teacher's teaching method, and current time period when the teacher's speech content involves the previous related content;

[0036] The test content of the "emotional attitude and value target" indicator: teaching courseware, teacher classroom teaching video recording. Extract the teacher's classroom teaching content, the teacher's position and the teacher's teaching method, and test whether the teacher's words contain positive emotions, attitudes and values; determine the teacher's position, teaching method, and current time period when the teacher teaches the content.

[0037] (2) Construct a standard library for detecting classroom teaching objectives and content. Construct a standard library for detecting the teaching characteristics of teachers’ classroom teaching objectives and use the content of the standard library as the benchmark for detecting the teaching characteristics of classroom teaching objectives.

[0038] The specific steps include:

[0039] (2-1) Establish a teaching goal content library. Collect multimodal data and information related to teaching goals. This includes but is not limited to teaching syllabi and multimodal data of teachers’ classroom teaching, such as text, images, audio, and video.

[0040] (2-2) Data preprocessing and standardization. Preprocess the collected data to collect feature information related to classroom teaching objectives. Perform word segmentation and stop word removal on text data. Perform speech segmentation on audio data to divide it into data of different speakers. Perform key frame extraction or convert video data into sequence data.

[0041] (2-3) Construct a standard library for classroom teaching goal content detection. Integrate the preprocessed data and construct an updateable classroom teaching goal content detection standard library Goal standard .

[0042] (3) Establish a method for detecting the characteristics of teachers’ classroom teaching objectives. Based on the teacher classroom teaching objective indicator system and the classroom teaching objective content detection standard library, establish a classroom teaching objective feature recognition algorithm and an indicator detection algorithm.

[0043] The specific steps include:

[0044] (3-1) Construct a feature recognition algorithm for multimodal classroom teaching data. Based on resource types such as teaching syllabus, teaching courseware, teacher classroom teaching video recordings, and classroom teaching target content detection standard library, the data types that need to be processed are divided into two types: text and video.

[0045] First, we build a speaker log confirmation model, and use it to extract the speech content of teachers and students in the video data as text data and save it. Then we build a Meta-Goal-Transformer model to comprehensively process the video data and text data, segment the video based on the time period obtained by the speaker log confirmation model, and then input the segmented video clips and text data into the Meta-Goal-Transformer model, and use Meta-Goal-Transformer to detect and classify the image frames of the video. The specific contents of the speaker log confirmation model and the Meta-Goal-Transformer model are as follows:

[0046] (3-1-1) Speaker log confirmation model. Speaker log confirmation is a framework based on the segmentation-clustering module. It consists of four parts. First, the input audio is detected using voice activity endpoints to remove non-human voices. Then the audio is segmented. Finally, the speaker model is used to extract speaker features. Finally, the features are clustered to identify the audio segments of each person and output the corresponding time information. The specific process is as follows: Figure 2 shown.

[0047] Speaker log confirmation model training: Use the CAM++ network to train the speech data in the content detection standard library to generate a speaker log confirmation model based on classroom teaching speech.

[0048] (3-1-2) Meta-Goal-Transformer model. Meta-Goal-Transformer uses a set of parameters to simultaneously encode classroom data from different modalities. It mainly consists of three parts: a multimodal data serializer for teacher classroom teaching goal teaching features, a multimodal shared encoder for teacher classroom teaching goal teaching features (Meta-Goal-Transformer encoder), and a prediction end for specific tasks. Its structure is as follows: Figure 3 shown.

[0049] Multimodal data serializer for teacher classroom teaching objectives and teaching features: converts the text modality and video modality data of teacher classroom teaching objectives and teaching features into different token embeddings, whose structure is as follows Figure 4 As shown in Figure 2, for text modal data, the original teaching content words are divided into subwords, and each subword is projected into the high-dimensional feature space of word embedding. Therefore, each input classroom teaching target text is converted into a set of x∈R n×D , where n is the number of tokens and D is the dimension of Embedding. For video modality data, the visual processing scheme for classroom teaching target feature detection first converts the input classroom teaching video x∈R H×W×C Converted into a sequence of 3Dpatches Where (H, W) represents the resolution of the original video, C represents the number of channels, S represents the Patch Size, and N s =(HW / S 2 ) represents the number of patches, and then a projection layer is used to project the embedding dimension to D dimensions:

[0050]

[0051] Meta-Goal-Transformer Encoder: ViT is used as the backbone network and pre-trained on the LAION-2B dataset through contrastive learning to enhance its ability to encode general tokens. For the text understanding of classroom teaching goals, CLIP's pre-trained text tokenizer is used to segment sentences into subwords and further convert them into word embeddings.

[0052] At the same time, add a learnable token x CLS , and obtain its final hiddenstate through the Transformer model It is usually used to perform classroom teaching target recognition tasks. In order to strengthen the position information, the position bi encoding is merged into the token Embedding and input to the Encoder. In addition, a learnable 1D position encoding is used. The architecture of TransformerEncoder follows the design of ViT:

[0053]

[0054] Where E x represents token Embedding, n represents the number of tokens, E pos Represents positional encoding.

[0055] Prediction side: After being represented by the Transformer Encoder, the teacher's classroom teaching goal data of different modalities are further sent to the head of the corresponding task. These heads are mainly composed of MLPs and vary according to the modality and task. The learning goal of Meta-Goal-Transformer can be summarized as follows:

[0056]

[0057] Among them, f(x), g(x), h(x) represent tokenizer, encoder and head respectively.

[0058] Language feature detection task training: Based on the general language understanding evaluation, we use the text dataset in the content detection standard library, replace the words in the text or add noise to the text without changing the main idea of ​​the sentence, and randomly disrupt the order of some sentences in order to achieve data enhancement to train the model's text matching task.

[0059] Image feature detection task training: Use Mask R-CNN as the detector and the student expression dataset in the content detection standard library to train the Meta-Goal-Transformer model for the target detection task.

[0060] (3-2) Construct a teacher's classroom teaching goal teaching feature index detection algorithm. Based on the features identified in (3-1), use the teaching goal feature detection algorithm to detect the identified features.

[0061] (3-2-1) Teaching goal feature detection algorithm. Based on the features obtained by the speaker log confirmation model and the Meta-Goal-Transformer model, combined with the classroom teaching goal detection content indicators, a detailed measurement method is further determined, as follows:

[0062] Speaker log confirmation: Use the speaker log confirmation model to extract the voice in the classroom video and confirm the speaker to form a speaker log information dictionary Dict Speak , the details are as follows:

[0063]

[0064] Among them, speaker represents the speaker, s represents the start time of the speech, e represents the end time of the speech, and content represents the speech content.

[0065] Teacher classroom teaching goal teaching feature indicator detection: First, use the language feature detection module of the Meta-Goal-Transformer model to detect the speaker log information dictionary Dict Speak The teacher's speech content is detected, and the teacher's teaching content summary is obtained. The obtained teaching content summary is compared with Goal standard The matching degree of the teaching goal content is calculated to obtain the teacher's classroom teaching video and Goal standard Content matching set Speaker log information dictionary Dict Speak Mark and classify the classroom teaching objectives mentioned in the teacher's speech content, and save the video according to time i , and pass the saved video clips to the Meta-Goal-Transformer model for image feature detection to obtain the detection results of the video data. Using the image feature detection of the Meta-Goal-Transformer model, for each classroom teaching goal teaching feature related video V i The position of the teacher in the classroom and the teaching method used are tested. The teaching methods mainly include narration, demonstration, and discussion. The position of the teacher mainly includes the podium and among the students. The teaching characteristics set of the teacher's classroom teaching objectives is constructed. in Details are as follows:

[0066]

[0067] Among them, time represents the teaching time period, type represents the category of classroom teaching objective indicators (where 1 represents knowledge and skill objectives, 2 represents process and method objectives, and 3 represents emotion, attitude and value objectives), way represents the teacher's teaching method (where 1 represents lectures, 2 represents demonstrations, and 3 represents discussions), and local represents the teacher's location (where 1 represents the podium and 2 represents among the students).

[0068] (4) Detect the teaching characteristics of teachers' classroom teaching objectives. Read the teacher's teaching resources to be detected, apply the teacher's classroom teaching objective teaching characteristics detection method based on the classroom teaching objective detection content and the classroom teaching objective content detection standard library, calculate the relevant data of each indicator, form the comprehensive detection results of the teacher's classroom teaching objective teaching characteristics, and feed back the detection results to the teacher. The specific steps include the following:

[0069] (4-1) Detection index-related data items. Extract the teacher teaching resources to be detected from the database, use the indicator feature detection algorithm to detect the teaching resources to be detected, and obtain the indicator feature results.

[0070] (4-2) Detect the teaching characteristics of the teacher's classroom teaching objectives. Count the results of the classroom teaching objectives teaching characteristics content detection, sort the teacher's classroom teaching objectives teaching characteristics set based on the time series, and draw a statistical chart of the teacher's classroom teaching objectives teaching characteristics in each time period of the class based on the sorting results, and obtain the teacher's teaching objectives teaching characteristics in the class; at the same time, based on the current results and the previous teacher's classroom teaching objectives teaching characteristics detection results, generate the teacher's classroom teaching objectives teaching characteristics of the time series, use the spectral analysis method to judge the teaching methods used for different types of classroom teaching objectives, and obtain the comprehensive detection results of the teacher's classroom teaching objectives teaching characteristics.

[0071] (4-3) Test feedback. Based on the test results of the teacher’s classroom teaching goal and teaching characteristics, the teacher’s classroom teaching goal and teaching characteristics are fed back to the teacher in the form of pictures and texts, so that the teacher can understand the teaching characteristics of his or her own classroom teaching goals and make innovations in future teaching. The feedback results are as follows: Figure 5 shown.

[0072] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0073] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting the characteristics of teacher classroom teaching objectives based on multimodal data, characterized in that The method comprises the following steps: (1) Determine the content of classroom teaching goal testing; according to the process and characteristics of course teaching goals, establish a teacher classroom teaching goal indicator system, and determine the specific testing content of teacher classroom teaching goal teaching characteristics testing; (2) Construct a standard library for detecting classroom teaching objectives and content; construct a standard library for detecting the characteristics of teachers’ classroom teaching objectives and use the contents of the standard library as a benchmark for detecting the characteristics of classroom teaching objectives and teaching; (3) Establish a method for detecting the teaching characteristics of teachers' classroom teaching objectives; establish a classroom teaching objective feature recognition algorithm and an indicator detection algorithm based on the teacher's classroom teaching objective indicator system and the classroom teaching objective content detection standard library.

2. The method for detecting the teaching characteristics of teacher classroom teaching objectives based on multimodal data according to claim 1 is characterized in that The following steps are also included: (4) Detect the teaching characteristics of teachers' classroom teaching objectives; read the teaching resources of the teachers to be tested, apply the detection method of teachers' classroom teaching objectives based on the classroom teaching objective detection content and the classroom teaching objective content detection standard library, calculate the relevant data of each indicator, form a comprehensive detection result of teachers' classroom teaching objective teaching characteristics, and feed back the detection result to the teachers.

3. The method for detecting the teaching characteristics of teacher classroom teaching objectives based on multimodal data according to claim 1 is characterized in that: The specific process of determining the classroom teaching target detection content in step (1) is as follows: (1-1) Determine the content indicators of classroom teaching objectives; from the perspective of teachers’ teaching objectives, determine that teachers’ teaching should have the following characteristics: To enable students to acquire basic knowledge or basic abilities, including the indispensable process and methods for solving problems, and to convey positive emotional values ​​to students, based on the above characteristics, the classroom teaching objectives of teachers are determined to include: knowledge and skills objectives, process and method objectives, emotional attitude and value objectives. The specific meaning and content of each indicator are as follows: "Knowledge and Skills Goals" indicator: used to evaluate whether teachers impart core knowledge or subject basic knowledge required for survival to students during classroom teaching, and whether they enable students to have information collection and processing capabilities, innovative spirit and practical ability, and lifelong learning ability; "Process and method objectives" indicator: used to evaluate whether teachers impart learning experiences and methods to students during classroom teaching, including thinking methods and learning methods in the learning process; "Emotional attitude and value goals" indicator: used to evaluate whether teachers convey positive emotional experiences, attitude tendencies and correct values ​​to students during classroom teaching; (1-2) Determine the content of the indicator test; according to the meaning of each indicator, determine the relevant test content, as follows: The test content of the "knowledge and skills objectives" indicator: teaching syllabus, teacher classroom teaching video recording; extract the teacher's classroom teaching content, teacher's position and teacher's teaching method; Calculate the degree of match between the teacher's speech content and the student knowledge and skill learning objectives proposed in the syllabus; determine the teacher's location, teaching method, and current time period when teaching the content related to this indicator; The test content of the "process and method objectives" indicator: teaching syllabus, teacher classroom teaching video recording; extracting the teacher's speech content and the teacher's position in the classroom teaching; calculating the degree of match between the teacher's speech content and the student learning process and method objectives proposed in the teaching syllabus; calculating the teacher's position, teacher's teaching method, and current time period when the teacher's speech content involves the previous related content; The testing content of the "emotional attitude, value and goal" indicator includes: teaching courseware, teacher's classroom teaching video recording; extracting the teacher's discourse content, the teacher's position and the teacher's teaching method in classroom teaching, and testing whether the teacher's discourse content contains positive emotions, attitudes and values; determining the teacher's position, teaching method and current time period when the teacher teaches the content.

4. The method for detecting the teaching characteristics of classroom teaching objectives based on multimodal data according to claim 1 is characterized in that: The specific process of step (2) of constructing a classroom teaching target content detection standard library is as follows: (2-1) Establish a teaching goal content library; collect multimodal data and information related to teaching goals, including teaching syllabus and multimodal data of teachers’ classroom teaching; (2-2) Data preprocessing and standardization: preprocess the collected data, collect feature information related to classroom teaching objectives, perform word segmentation and stop word removal on text data; segment audio data into different speaker data; extract key frames from video data or convert them into sequence data; (2-3) Construct a standard library for classroom teaching content testing; Integrate the preprocessed data and build an updateable classroom teaching goal content detection standard library Goal standard .

5. The method for detecting the teaching characteristics of classroom teaching objectives based on multimodal data according to claim 1 is characterized in that: The specific process of step (3) of establishing a method for detecting the characteristics of classroom teaching objectives is as follows: (3-1) Construct a feature recognition algorithm for multimodal classroom teaching data; divide the data types that need to be processed into two types: text and video, based on the teaching syllabus, teaching courseware, teacher classroom teaching video resource types, and classroom teaching target content detection standard library; First, a speaker log confirmation model is built. The speaker log confirmation model is used to extract the speech content of teachers and students in the video data as text data and save it. Then, a Meta-Goal-Transformer model is built to comprehensively process the video data and text data. The video is segmented based on the time period obtained by the speaker log confirmation model. The segmented video clips and text data are then input into the Meta-Goal-Transformer model. The Meta-Goal-Transformer is used to detect and classify the image frames of the video. The specific contents of the speaker log confirmation model and the Meta-Goal-Transformer model are as follows: (3-1-1) Speaker log confirmation model. Speaker log confirmation is a framework based on the segmentation-clustering module. It consists of four parts. First, the input audio is subjected to speech activity endpoint detection to remove non-human voices. Then the audio is segmented. Finally, the speaker model is used to extract speaker features. Finally, the features are clustered to identify the audio segment of each person and output the corresponding time information. Speaker log confirmation model training: Use the CAM++ network to train the speech data in the content detection standard library to generate a speaker log confirmation model based on classroom teaching speech; (3-1-2) Meta-Goal-Transformer model. Meta-Goal-Transformer uses a set of parameters to simultaneously encode classroom data from different modalities. It consists of three parts: a multimodal data serializer of teacher classroom teaching goal teaching features, a multimodal shared encoder of teacher classroom teaching goal teaching features (Meta-Goal-Transformer encoder), and a prediction end for specific tasks. Teacher classroom teaching objectives teaching features multimodal data serializer: The text modality and video modality data of the teacher classroom teaching objectives teaching features are converted into different token embeddings. For text modality data, the original teaching content words are divided into subwords, and each subword is projected into the high-dimensional feature space of word embedding. Therefore, each input classroom teaching objective text is converted into a set of x∈R n×D , where n is the number of tokens and D is the dimension of Embedding. For video modality data, the visual processing scheme for classroom teaching target feature detection first converts the input classroom teaching video x∈R H×W×C Converted into a sequence of 3D patches Where (H, W) represents the resolution of the original video, C represents the number of channels, S represents the Patch Size, and N s =(HW / S 2 ) represents the number of patches, and then a projection layer is used to project the embedding dimension to D dimensions: Meta-Goal-Transformer encoder: Use ViT as the backbone network and pre-train it on the LAION-2B dataset through contrastive learning to strengthen its ability to encode general tokens. For the text understanding of classroom teaching goals, use CLIP's pre-trained text tokenizer to segment sentences into subwords and further convert them into word embeddings; At the same time, a learnable token x is added CLS , and obtain its final hidden state through the Transformer model It is usually used to perform classroom teaching target recognition tasks. In order to strengthen the position information, the position bi encoding is merged into the tokenEmbedding and input to the Encoder. In addition, a learnable 1D position encoding is used. The architecture of the Transformer Encoder follows the design of ViT: z l ′ =MSA(LN(z l-1 ))+z l-1 ,l=1...L With l =MLP(LN(z l ′ ))+z l ′ ,l=1...L Where E x represents token Embedding, n represents the number of tokens, E pos Indicates positional encoding; Prediction side: After being represented by the Transformer Encoder, the teacher's classroom teaching goal data of different modalities are further sent to the head of the corresponding task. These heads are composed of MLPs and vary according to the modality and task. The learning goal of Meta-Goal-Transformer can be summarized as follows: Among them, f(x), g(x), h(x) represent tokenizer, encoder and head respectively; Language feature detection task training: Based on the general language understanding evaluation, we use the text dataset in the content detection standard library, replace the words in the text or add noise to the text without changing the main idea of ​​the sentence, and randomly disrupt the order of some sentences to achieve data enhancement to train the model's text matching task; Image feature detection task training: Use Mask R-CNN as the detector and the student expression dataset in the content detection standard library to train the Meta-Goal-Transformer model for the target detection task; (3-2) Construct a teacher's classroom teaching goal teaching feature index detection algorithm; Based on the features identified in (3-1), use the teaching goal feature detection algorithm to detect the identified features; (3-2-1) Teaching goal feature detection algorithm, based on the features obtained by the speaker log confirmation model and the Meta-Goal-Transformer model, combined with the classroom teaching goal detection content indicators, further determines the detailed measurement method, as follows: Speaker log confirmation: Use the speaker log confirmation model to extract the voice in the classroom video and confirm the speaker to form a speaker log information dictionary Dict Speak , the details are as follows: Among them, speaker represents the speaker, s represents the start time of speech, e represents the end time of speech, and content represents the content of speech; Teacher classroom teaching goal teaching feature indicator detection: First, use the language feature detection module of the Meta-Goal-Transformer model to detect the speaker log information dictionary Dict Speak The teacher's speech content is detected, and the teacher's teaching content summary is obtained. The obtained teaching content summary is compared with Goal standard The matching degree of the teaching goal content is calculated to obtain the teacher's classroom teaching video and Goal standard Content matching set Speaker log information dictionary Dict Speak Mark and classify the classroom teaching objectives mentioned in the teacher's speech content, and save the video according to time i , and pass the saved video clips to the Meta-Goal-Transformer model for image feature detection, obtain the detection results of the video data, use the image feature detection of the Meta-Goal-Transformer model, and perform a video V related to each classroom teaching goal teaching feature. i The position of the teacher and the teaching method used in the classroom are tested. The teaching methods include lectures, demonstrations, and discussions. The positions of the teacher include the podium and among the students. The teaching characteristics set of the teacher's classroom teaching objectives is constructed. in Details are as follows: Among them, time represents the teaching time period, type represents the category of classroom teaching objective indicators, where 1 represents knowledge and skill objectives, 2 represents process and method objectives, and 3 represents emotion, attitude and value objectives; way represents the teacher's teaching method, where 1 represents lecture, 2 represents demonstration, and 3 represents discussion; local represents the teacher's location, where 1 represents the podium and 2 represents among the students.

6. The method for detecting the characteristics of teacher classroom teaching objectives based on multimodal data according to claim 2 is characterized in that: The specific process of step (4) detecting the teaching characteristics of the teacher's classroom teaching objectives is as follows: (4-1) Detecting index-related data items, extracting the teacher teaching resources to be detected from the database, using the index feature detection algorithm to detect the teaching resources to be detected, and obtaining the index feature results; (4-2) Detect the teaching characteristics of the teacher's classroom teaching objectives, count the detection results of the teaching characteristics of the classroom teaching objectives, sort the teaching characteristics of the teacher's classroom teaching objectives based on the time series, and draw a statistical graph of the teaching characteristics of the teacher's classroom teaching objectives in each time period of the class based on the sorting results, and obtain the teaching characteristics of the teacher's classroom teaching objectives in the class; at the same time, based on the current results and the previous detection results of the teaching characteristics of the teacher's classroom teaching objectives, generate the teaching characteristics of the teacher's classroom teaching objectives in the time series, use the spectral analysis method to judge the teaching methods used for different types of classroom teaching objectives, and obtain the comprehensive detection results of the teaching characteristics of the teacher's classroom teaching objectives; (4-3) Detection feedback: Based on the detection results of the teacher’s classroom teaching objectives and teaching characteristics, the teacher’s classroom teaching objectives and teaching characteristics are fed back to the teacher in the form of pictures and texts.

7. A teacher classroom teaching objective teaching feature detection system based on multimodal data, characterized in that: The method for detecting the teaching characteristics of the teacher's classroom teaching objectives based on multimodal data as described in any one of claims 1 to 6 comprises: The indicator detection content module confirms the specific detection content of each indicator and the model to be used; The classroom teaching goal content detection standard library module uses the language feature detection model to extract the classroom teaching goal content in the teaching syllabus to form a classroom teaching goal content detection standard library; Multimodal data processing module, building and training a teacher classroom teaching objective teaching feature detection model for multimodal data; The speaker log confirmation module processes the voice data, gives each person's audio segment, and outputs the corresponding time information; Model training module, based on the model network of multimodal data processing, trains the feature detection model for teachers’ classroom teaching objectives; The teacher's classroom teaching objectives and teaching characteristics detection module processes the detection results of various indicators and conducts a comprehensive detection of the teacher's classroom teaching objectives and teaching characteristics.

Images