Method, system and terminal for supervising polymorphic identification teaching behaviors

Abstract

The invention belongs to the field of teaching, and discloses a supervision method, system and terminal for polymorphic identification of teaching behaviors, and the method specifically comprises the steps: obtaining a to-be-determined classroom media stream, determining the speech characteristics of a teacher and the speech characteristics of a student according to the classroom media stream, and determining the classroom activity according to the speech characteristics of the teacher and the speech characteristics of the student. According to the method, the speech features of the teacher and the students in the classroom are extracted from the collected classroom media stream, and the classroom enthusiasm of the students is analyzed according to the speech features of the teacher and the students, so that the teacher can adjust the teaching mode timely, and the activeness of the students in class is aroused.

Description

Technical Field

[0001] The present application relates to the field of teaching, and in particular to a supervision method, system and terminal for multi-modal recognition of teaching behaviors. Background Art

[0002] With the development of sensor technology, the school teaching environment and teachers' teaching methods have also changed accordingly. Classroom learning is more interactive, the teaching of knowledge concepts is more concrete, and the teaching methods are more personalized.

[0003] In the traditional teaching model, students passively accept knowledge through teachers' teaching, which causes most students to gradually develop a numb habit of not wanting to ask "why" and not knowing how to ask "why", thus forming a mindset of blindly worshipping books and teachers. This learning method not only restricts the development of students' thinking, but also gradually makes students lose their initiative in learning, and even forces them to learn, without experiencing the joy of learning at all.

[0004] In view of the existing teaching environment, teachers need to adjust their teaching methods in a timely manner and actively mobilize students' enthusiasm. Therefore, how to let teachers adjust their teaching methods in a timely manner, mobilize students' enthusiasm in class, and urge students to improve their class efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of this application is to provide a supervision method for multi-modal identification of teaching behaviors, which can help teachers adjust teaching methods in a timely manner, mobilize students' enthusiasm in class, and urge students to improve their class efficiency.

[0006] In a first aspect, an embodiment of the present application provides a supervision method for multi-modal recognition of teaching behaviors, the method comprising:

[0007] Get the classroom media stream to be determined;

[0008] Determining the teacher's speech and behavior characteristics and the student's speech and behavior characteristics according to the classroom media stream;

[0009] Determine the level of classroom activity based on the characteristics of teachers' and students' words and deeds.

[0010] In an achievable embodiment, the speech and behavior features include actions and voices; and determining the teacher's speech and behavior features and the student's speech and behavior features according to the classroom media stream includes:

[0011] Extracting actions in the video and sounds in the audio of the media stream, wherein the actions include teacher actions and student actions, and the sounds include teacher voices and student voices;

[0012] Determine the teacher's actions and voice based on the actions in the video and the sounds in the audio, and extract the teacher's speech and behavior features;

[0013] Determine student actions and voices based on the actions in the video and the sounds in the audio, and extract student speech and behavior features.

[0014] In an achievable embodiment, determining the teacher's actions and voice according to the actions in the video and the voice in the audio, extracting the teacher's speech and behavior features, determining the student's actions and voice according to the actions in the video and the voice in the audio, extracting the student's speech and behavior features, includes:

[0015] According to a preset face information database, combined with the face information in the media stream, the position of the teacher and the position of each student are determined, and the human body movements in the area where each teacher is located and the human body movements in the area where each student is located are intercepted to obtain the behavioral characteristics of each teacher and each student; the human body movements include lip movement dynamics and torso movements;

[0016] Determine the speech of each teacher based on the lip movement dynamics captured in the area where each teacher is located and the audio in the media stream to obtain the speech features of each teacher;

[0017] According to the lip movement dynamics intercepted in the location area of ​​each student and the audio in the media stream, the voice uttered by each student is determined to obtain the voice features of each student.

[0018] In an achievable embodiment, the method of determining the voice emitted by each teacher based on the lip movement dynamics intercepted in the location area of ​​each teacher and the audio in the media stream to obtain the voice features of each teacher, and determining the voice emitted by each student based on the lip movement dynamics intercepted in the location area of ​​each student and the audio in the media stream to obtain the voice features of each student includes:

[0019] According to the preset lip movement dynamic library, the estimated speech corresponding to the lip movement dynamics of each teacher and each student is determined;

[0020] Parsing the audio in the media stream to generate multiple segments of actual speech;

[0021] Match each of the estimated voices with the actual voice, determine the actual voice corresponding to each lip movement, and use the actual voice corresponding to each lip movement as the voice feature of the corresponding person.

[0022] In an achievable embodiment, determining the classroom activity according to the teacher's speech and behavior characteristics and the student's speech and behavior characteristics includes:

[0023] According to a preset interactive speech and behavior feature database between teachers and students, combining the speech and behavior features of each teacher and the speech and behavior features of each student, determining the interactive speech and behavior features of each teacher and all the speech and behavior features of each student;

[0024] For each teacher and student, the activity level of each teacher and each student is determined based on the proportion of the corresponding interactive speech and behavior characteristics to the corresponding total speech and behavior characteristics.

[0025] In an achievable embodiment, the interactive speech and behavior features of each teacher and each student are determined based on a preset database of interactive speech and behavior features between teachers and students, combined with the speech and behavior features of each teacher and the speech and behavior features of each student, and the interactive speech and behavior features of each teacher and each student include:

[0026] According to the preset database of interactive speech and behavior characteristics between teachers and students, the similarity between the interactive speech and behavior characteristics of each teacher and each student is determined by combining the speech and behavior characteristics of each teacher and the speech and behavior characteristics of each student;

[0027] Determining whether the similarity of the interactive speech and behavior features is greater than a preset threshold;

[0028] If the speech and behavior characteristics are judged to be interactive speech and behavior characteristics, otherwise they are teacher speech and behavior characteristics and student speech and behavior characteristics respectively.

[0029] In an achievable embodiment, the method of determining the similarity of the interactive speech and behavior features of each teacher and each student based on a preset database of interactive speech and behavior features between teachers and students, combining the speech and behavior features of each teacher and the speech and behavior features of each student, comprises:

[0030] The joint position coordinates of each teacher and each student's actual posture are taken as the actual posture vector;

[0031] The posture similarity is determined based on the distance between the actual posture vector and the posture vectors of each teacher and each student in the interactive speech and behavior feature database.

[0032] In an achievable embodiment, the similarity of speech and behavior features includes: semantic similarity, and the similarity of the interactive speech and behavior features of each teacher and each student is determined based on the preset interactive speech and behavior feature database of teachers and students, combined with the speech and behavior features of each teacher and the speech and behavior features of each student, and further includes:

[0033] Parse the audio of each teacher and each student, determine the phonetic text, and extract the actual words in the phonetic text;

[0034] The speech similarity is determined based on the number of identical actual words with the semantic words of each teacher and each student in the interactive speech and behavior database.

[0035] In a second aspect, the present application also provides a supervision system for multi-modal recognition of teaching behaviors, which has the function of implementing the method in the first aspect or any possible implementation thereof. Specifically, the system includes a unit for implementing the method in the first aspect or any possible implementation thereof.

[0036] In one embodiment, the system includes:

[0037] The acquisition module obtains the classroom media stream to be determined;

[0038] A processing module, for determining the teacher's speech and behavior characteristics and the student's speech and behavior characteristics according to the classroom media stream;

[0039] The activity determination module determines the classroom activity based on the characteristics of the teacher's words and deeds and the students' words and deeds.

[0040] In a third aspect, the present application also provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the computer program.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above items is implemented.

[0042] In a fifth aspect, the present application further provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute any one of the methods of the first aspect.

[0043] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0044] This method extracts the speech and behavior characteristics of teachers and students in the classroom from the collected classroom media stream, and analyzes the students' classroom enthusiasm based on the speech and behavior characteristics of teachers and students, which helps teachers adjust teaching methods in a timely manner, urge students to improve classroom efficiency, and mobilize students' enthusiasm in class. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 An application environment diagram of a supervision method for multi-modal recognition of teaching behaviors provided in this application;

[0047] Figure 2 A schematic diagram of a mobile monitoring page for a supervision method for multi-modal recognition of teaching behaviors provided in this application;

[0048] Figure 3 A flowchart of a supervision method for multi-modal recognition of teaching behaviors provided in this application;

[0049] Figure 4 A flowchart for extracting the speech and behavior features of teachers and students provided for this application;

[0050] Figure 5 A flowchart of obtaining teacher and student voice features provided for this application;

[0051] Figure 6 A schematic diagram of the process of matching teacher and student voice features provided for this application;

[0052] Figure 7 A flowchart for determining classroom activity based on the speech and behavior characteristics of teachers and students provided for this application;

[0053] Figure 8 A flowchart for determining the speech and behavior characteristics of each teacher and the interactive speech and behavior characteristics among all speech and behavior characteristics of each student provided for this application;

[0054] Fig. 9 A schematic diagram of a similarity process for determining the interactive speech and behavior characteristics of each teacher and each student provided for this application;

[0055] Fig.10 A schematic diagram of a student raising his hand in an embodiment provided by the present application;

[0056] Fig.11 A schematic diagram of another student raising his hand in an embodiment provided by the present application;

[0057] Fig.12 Another similarity flow diagram for determining the interactive speech and behavior characteristics of each teacher and each student provided for this application;

[0058] Fig.13 A diagram of the pre-class communication between teachers and students provided for this application;

[0059] Fig.14 Diagram of teacher-student communication in class provided for this application;

[0060] Fig.15 Schematic diagram of after-class communication between teachers and students provided for this application;

[0061] Fig.16 A schematic diagram of the structure of a supervision system for multi-modal recognition of teaching behaviors provided in this application;

[0062] Fig.17 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0063] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, systems, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0064] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0065] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0066] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0067] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0068] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0069] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0070] Figure 1 This is an application environment diagram of a multi-modal teaching behavior supervision method provided by this application. This method can be used to monitor and count classroom activity in real time, such as Figure 1 As shown in the figure, during class, you can choose to open the real-time monitoring software of students' classroom activity at the edge of the smart blackboard. You can make real-time statistics on comprehensive classroom indicators such as students' classroom sign-in situation, front row seating rate, classroom answer accuracy rate, head-up rate, etc., which is convenient for teachers to adjust the teaching mode in real time. At the same time, the detection software can also display and monitor the classroom links, which is convenient for teachers to grasp the teaching rhythm. This method is applied to student classrooms. Not only can teachers monitor the classroom atmosphere on the blackboard, but they can also review the classroom situation through mobile phones after class. Figure 2 As shown in , the file selected for playback on the mobile phone is part of the courseware in class, and the teacher can adaptively adjust his or her teaching method through video playback. In addition, in order to obtain the media stream required by this method, such as Figure 1 As shown in the figure, the camera above the smart blackboard records the teaching video, the microphone on the podium collects the ambient voice, and the sensor inside the smart blackboard can obtain the teacher's writing action on the blackboard. The method obtains the classroom media stream to be determined, and then determines the teacher's speech and behavior characteristics and the student's speech and behavior characteristics according to the classroom media stream, and finally determines the classroom activity according to the teacher's speech and behavior characteristics and the student's speech and behavior characteristics. Through this method, teachers can not only urge students to improve their class efficiency, but also adjust the teaching plan in time, mobilize students' classroom activity, and the post-class review link is also conducive to teachers improving their teaching level.

[0071] Figure 3 This is a flowchart of a supervision method for multi-modal recognition of teaching behaviors provided by this application.

[0072] S301, obtaining a classroom media stream to be determined.

[0073] S302, determining the teacher's speech and behavior characteristics and the student's speech and behavior characteristics according to the classroom media stream.

[0074] S303, determining the classroom activity level according to the teacher's speech and behavior characteristics and the student's speech and behavior characteristics.

[0075] Exemplarily, the classroom environment is captured by the camera of the smart blackboard, multiple cameras in the classroom, and the sound collection equipment on the podium to obtain the classroom media stream, and the speech and behavior characteristics of teachers and students are determined according to the classroom media stream. The teacher's speech and behavior characteristics include: when the teacher interacts with students, he will stand around the interactive students, when asking students to answer questions, he will raise his hands, and when preparing to interact with students, he will say "...Is that right?" or "Please ask classmate xxx to answer it." The student's speech and behavior characteristics include: when students answer questions, they will stand up, when students interact with students, they will talk face to face, and when students have questions about the explanation, they will raise their hands. The above-mentioned teacher and student speech and behavior characteristics are compared with the speech and behavior in the preset speech and behavior feature library, and the speech and behavior that conforms to the teacher-student interaction and student-student interaction is screened out. The proportion of interactive speech and behavior in the entire media stream to be determined is calculated to determine the activity of the classroom.

[0076] Through this method, teachers can not only monitor students' class activity in real time and urge students to improve their class efficiency, but also facilitate teachers' post-class review, which can remind teachers to change their teaching plans in time and further mobilize students' class activity.

[0077] In an embodiment that can be implemented in the present application, the speech and behavior features include actions and voices; Figure 4 As shown, the determining of the teacher's speech and behavior characteristics and the student's speech and behavior characteristics according to the classroom media stream includes:

[0078] S401, extracting actions in the video and sounds in the audio of the media stream, wherein the actions include teacher actions and student actions, and the sounds include teacher voices and student voices.

[0079] S402, determining the teacher's actions and voice according to the actions in the video and the sounds in the audio, and extracting the teacher's speech and behavior features.

[0080] S403, determining the student's actions and voices according to the actions in the video and the voices in the audio, and extracting the student's speech and behavior features.

[0081] Exemplarily, extract student actions and teacher actions in the video, extract student voices and teacher actions in the audio, for example, teacher actions include: teacher writing actions on the smart blackboard, gestures when explaining problems, and teacher walking movements, and teacher voices include: teacher speaking speed, intonation, and expression. Student actions include students raising their hands, writing, and reading, and student voices include: students asking questions, answering, and discussing. By extracting student and teacher actions and voices, analyze the characteristics of teachers and students' words and deeds, and judge whether students actively participate in classroom discussions or actively respond to questions raised by teachers.

[0082] Exemplarily, the teacher's speech and behavior characteristics include: when the teacher makes a sound but no student stands up to interact, it is defined as the teacher's teaching behavior; when the teacher speaks and the voice extracts a question, it is defined as the teacher asking a question; when the teacher uses a smart blackboard to write, it is defined as blackboard writing. The student's speech and behavior characteristics include: when the student stands up and the voice is recognized as speaking, it is defined as student speaking; when multiple students' voices are recognized without the teacher's voice in between, it is defined as student-student interaction; and when there is no voice in the teacher, it is defined as independent learning. This method combines the behavior and voice of students and teachers to judge the speech and behavior characteristics of students and teachers at this time, thereby realizing the distinction between speech and behavior characteristics.

[0083] The analysis of this method can determine students' classroom activity, help students and teachers understand their own classroom performance, and help further improve their classroom status.

[0084] In one embodiment that can be implemented in the present application, Figure 5 As shown, the teacher's actions and voices are determined according to the actions in the video and the voices in the audio, and the teacher's speech and behavior features are extracted; the student's actions and voices are determined according to the actions in the video and the voices in the audio, and the student's speech and behavior features are extracted, including:

[0085] S501, according to a preset face information database, combined with the face information in the media stream, determine the position of the teacher and the position of each student, and intercept the human body movements in the area where each teacher is located and the human body movements in the area where each student is located, to obtain the behavioral characteristics of each teacher and each student; the human body movements include lip movement dynamics and torso movements.

[0086] S502, determining the voice uttered by each teacher based on the lip movement dynamics captured in the location area of ​​each teacher and the audio in the media stream, and obtaining the voice features of each teacher.

[0087] S503, determining the voice uttered by each student based on the lip movement dynamics intercepted in the location area of ​​each student and the audio in the media stream, and obtaining the voice features of each student.

[0088] For example, extract a media stream from a math class, first determine the facial information in the media stream, combine it with the preset facial information database, determine that the teacher is at the podium and that each student is in his or her seat, and after obtaining the position information of everyone, intercept the body movements of the teacher at the podium and the body movements of the students in their seats. If the teacher has not only torso movements but also lip movements at the podium, then combine it with the audio in the media stream to determine the teacher's voice information and obtain the teacher's voice features. At this time, if the student has not only torso movements but also lip movements at his or her seat, then combine it with the audio in the media stream to determine the student's voice information and obtain the student's voice features.

[0089] This method can accurately obtain the voice features of each student and each teacher, further match the speech and behavior information of teachers and students, and improve the accuracy of determining students' classroom activity.

[0090] In the embodiments that can be implemented in this application, Figure 6 As shown, the speech emitted by each teacher is determined based on the lip movement dynamics intercepted in the location area of ​​each teacher and the audio in the media stream to obtain the speech features of each teacher, and the speech emitted by each student is determined based on the lip movement dynamics intercepted in the location area of ​​each student and the audio in the media stream to obtain the speech features of each student, including:

[0091] S601, determining the estimated speech corresponding to the lip movement dynamics of each teacher and each student according to a preset lip movement dynamic library;

[0092] S602, parsing the audio in the media stream to generate multiple actual voice segments;

[0093] S603, matching each of the estimated voices with the actual voices, determining the actual voices corresponding to each lip movement, and using the actual voices corresponding to each lip movement as the voice features of the corresponding person.

[0094] For example, suppose a Chinese teacher is teaching a Pinyin course in class. According to the lip movements detected in the video, the estimated voice is "a". The media stream corresponding to the Chinese class is parsed to generate the actual voice at the corresponding moment. The actual voice at this time is "a". Through matching, it is determined that the actual voice corresponding to the lip movements is "a", and it is used as the voice feature of the Chinese teacher.

[0095] For example, suppose the estimated speech corresponding to the student's lip movements in the video is "I haven't understood the solution to this problem yet", and the actual speech obtained by parsing the media stream is "I haven't understood the solution to this problem yet". Through matching, it is determined that the actual speech corresponding to the lip movements is "I haven't understood the solution to this problem yet", and it is used as the student's speech feature.

[0096] This method more accurately distinguishes the student's voice features from the teacher's voice features by matching the character's lip movements with the actual parsed speech.

[0097] In the embodiments that can be implemented in this application, Figure 7 As shown, the determination of classroom activity based on the teacher's speech and behavior characteristics and the student's speech and behavior characteristics includes:

[0098] S701, according to a preset interactive speech and behavior feature database between teachers and students, combining the speech and behavior features of each teacher and the speech and behavior features of each student, determining the interactive speech and behavior features of each teacher and all the speech and behavior features of each student;

[0099] S702, for each teacher and each student, determine the activity level of each teacher and each student according to the proportion of the corresponding interactive speech and behavior features to the corresponding total speech and behavior features.

[0100] It should be noted that the characteristics of each teacher's speech and behavior include: teacher lectures, teacher questions, teacher inspections and blackboard writing, and the characteristics of students' speech and behavior include: student reports, teacher-student interaction, student-student interaction and independent learning. Among them, teacher questions, student reports, teacher-student interaction and student-student interaction can all be counted as interactive speech and behavior characteristics of the classroom. The proportion of interactive speech and behavior characteristics in the entire speech and behavior characteristics can be used to judge the atmosphere of this class and the activity of students.

[0101] Exemplarily, each teacher's words and deeds are recorded as Nt, including: teacher lectures, teacher questions, teacher inspections and blackboard writing. According to the characteristics of each teacher's words and deeds, the length of each teacher's lectures, lectures, blackboard writing, average speaking speed, number of questions, and number of words in the lectures can also be counted. Each student's words and deeds are recorded as Ns, including: student speeches, teacher-student interactions, student-student interactions, and independent learning. The teacher's words and deeds occupancy rate in the classroom is recorded as Rt. The teacher's words and deeds occupancy rate is the number of teachers' words and deeds minus the number of words and deeds of all characters. The higher the teacher's words and deeds occupancy rate, the worse the overall classroom interaction. For example, in an English class, the teacher asked 10 questions, the teacher gave 2 lectures, the teacher patrolled 1 time, wrote on the blackboard 20 times, students answered questions 5 times, and students interacted 1 time. At this time, the teacher's words and deeds are recorded as 23 times, and the number of words and deeds of all people in the whole class is 29 times. The teacher's words and deeds occupancy rate Rt is 23 / 29. At this time, the classroom activity is approximately equal to 0.79, which basically shows that the whole English class is dominated by the words and deeds of the English teacher, and the students are not actively integrated into the classroom atmosphere. In addition, the activity of the classroom can be judged by calculating the words and deeds conversion rate Ch. The calculation formula of the words and deeds conversion rate Ch is as follows: Ch = (g-1) / N, where the Ch value is between 0-1, N is the sum of the words and deeds of all teachers and students, and the lower the Ch value, the more dull the classroom atmosphere is, and there is a lack of communication between teachers and students. For example, in a math class, after the teacher asked 10 questions, students answered 5 of them. After the students answered, the teacher continued to teach and explain. For the other 5 times, no one answered and the teacher continued to teach and explain. While asking questions and teaching, the teacher wrote on the blackboard 20 times. Among them, there were 6 times when students discussed with each other after writing on the blackboard. After the students discussed, the teacher continued to teach and explain. The number of switches between the teacher's words and deeds and the students' words and deeds is recorded as g. At this time, g=22 times. The number of times everyone spoke and did something in the whole class was 41 times. The conversion rate of words and deeds was calculated to be (22-1) / 41, which is 21 / 41. At this time, the classroom activity of the math class is approximately equal to 0.51, and the students' enthusiasm for learning is high.

[0102] Exemplarily, each teacher's words and deeds are recorded as Nt, including: teacher lectures, teacher questions, teacher inspections and blackboard writing. According to the characteristics of each teacher's words and deeds, the length of each teacher's teaching time, teaching time, blackboard writing time, average speaking speed, number of questions and number of words in the class can also be counted. Each student's words and deeds are recorded as Ns, including: student speeches, teacher-student interaction, student-student interaction and independent learning. The proportion of the teacher's teaching time, inspection time and blackboard writing time in each class to the total teaching time is calculated and recorded as the teacher's words and deeds time ratio Ft. The student's classroom activity is judged based on this ratio. The higher the ratio, the lower the classroom activity and the duller the classroom atmosphere. For example, an English class has a total teaching time of 45 minutes, and the total teaching time of the teacher, inspection time and blackboard writing time is 20 minutes. At this time, the teacher's words and deeds time ratio is 4 / 9, and the classroom activity at this time is approximately equal to 0.44.

[0103] Exemplarily, statistical charts are used to summarize the calculated classroom activity related data, for example, a classroom activity line chart, a classroom activity pie chart, etc.

[0104] When calculating the classroom activity, we can make judgments by calculating the teacher's words and actions occupancy rate, the teacher-student words and actions conversion rate, and the teacher's words and actions duration occupancy rate. This method realizes the digitization of classroom activity, which makes it more convenient and intuitive to understand the students' classroom situation.

[0105] In the embodiments that can be implemented in this application, Figure 8 As shown, the interactive speech and behavior features of each teacher and each student are determined based on the preset interactive speech and behavior feature database between the teacher and the student, combining the speech and behavior features of each teacher and the speech and behavior features of each student, and the interactive speech and behavior features of all the speech and behavior features of each teacher and each student include:

[0106] S801, according to a preset database of interactive speech and behavior features between teachers and students, combining the speech and behavior features of each teacher and the speech and behavior features of each student, determining the similarity of the interactive speech and behavior features of each teacher and each student;

[0107] S802, determining whether the similarity of the interactive speech and behavior features is greater than a preset threshold;

[0108] S803, if it is determined that the speech and behavior feature is an interactive speech and behavior feature, otherwise it is a teacher's speech and behavior feature and a student's speech and behavior feature respectively.

[0109] For example, it is assumed that the characteristics of the teacher's speech and behavior include: teaching courseware, guiding students, writing on the blackboard, initiating discussions, and patiently answering questions, and the characteristics of the student's speech and behavior include: answering the teacher's questions, writing homework, group discussion, and asking questions. When students and teachers interact, there will be actions or voices before the interaction. For example, when the teacher asks a question to the student, there will be a hand-raising gesture. When the teacher and students discuss a problem or the students discuss with each other, there will be face-to-face actions and corresponding voices. When the student asks a question, there will be a hand-raising action. If the student raises his hand and sends a question voice "Teacher, I have a question that I don't understand", the student's hand-raising action and question voice are matched with the standard interactive gestures and the semantic words that have been recorded in the interactive speech and behavior feature database. If the similarity between the two actions is greater than 60%, the student's hand-raising is used as an interactive speech and behavior feature. If the similarity between the intercepted voice and the recorded semantics is greater than 80%, the student's speech is used as an interactive speech and behavior feature.

[0110] This embodiment compares the actual speech and behavior features of teachers and students with the interactive speech and behavior features in the interactive speech and behavior feature database to more accurately match the interactive speech and behavior features of teachers and students.

[0111] In the embodiments that can be implemented in this application, Fig. 9 As shown, the similarity of speech and behavior features includes: posture similarity, and the similarity of the interactive speech and behavior features of each teacher and each student is determined based on the preset interactive speech and behavior feature database of teachers and students, combined with the speech and behavior features of each teacher and the speech and behavior features of each student, including:

[0112] S901, taking the joint position coordinates of the actual posture of each teacher and each student as the actual posture vector;

[0113] S902, determining posture similarity according to the distance between the actual posture vector and the posture vectors of each teacher and each student in the interactive speech and behavior feature database.

[0114] For example, when a student raises his hand in class, he will raise his left arm in his seat. Fig.10 The standard interactive posture of students raising their hands in the interactive speech and behavior feature database is recorded. However, in actual situations, students' movements are not necessarily very standard, and there is a certain deviation from the standard interactive posture recorded in the system. Fig.11 This is the actual hand-raising posture of the students. Obviously, there is a certain deviation between the actual hand-raising posture and the standard hand-raising posture. In order to accurately identify the gesture of the student requesting interaction, the vector distance between the actual fingertips and the fingertips of the standard interactive posture is calculated to determine the value of the deviation. When the vector distance is controlled within Fig.11When the actual posture is within the range shown in , it is determined to be the same posture, and the similarity between the actual posture and the standard interactive posture is determined by calculating the vector distance between each joint of the human body in the actual posture and each joint in the standard interactive posture.

[0115] This embodiment compares the actual action features of the teacher and the student with the standard interactive posture features in the interactive speech and behavior feature database to more accurately match the interactive action features of the teacher and the student.

[0116] In the embodiments that can be implemented in this application, Fig.12 As shown, the similarity of speech and behavior features includes: semantic similarity, and the similarity of the interactive speech and behavior features of each teacher and each student is determined based on the preset interactive speech and behavior feature database of teachers and students, combined with the speech and behavior features of each teacher and the speech and behavior features of each student, and further includes:

[0117] S1201, parsing the audio of each teacher and each student, determining the phonetic text, and extracting the actual words in the phonetic text;

[0118] S1202, determining the speech similarity according to the number of identical semantic words between the actual words and each teacher and each student in the interactive speech and behavior database.

[0119] For example, suppose there is a database of interactive speech and behavior features between a teacher and a student, which contains the following information:

[0120] The teacher's voice characteristics: guiding students to think, encouraging students to express their opinions, and providing examples to illustrate; the student's voice characteristics: actively participating in discussions, putting forward inspiring opinions, and giving examples to illustrate.

[0121] Now, suppose there is an audio clip containing the conversation between a teacher and a student. First, the audio can be parsed to convert the speech in the audio clip into text and extract the actual words. Then, the similarity of the speech is determined based on the number of actual words that are identical to the semantic words of each teacher and student in the interactive speech and behavior database. For example, suppose the actual words obtained after parsing the audio clip are: guide, student, think, encourage, participate, discuss, viewpoint, and give an example. For the teacher's speech features, the semantic words that are identical to the actual words are: guide, student, think, encourage, and viewpoint. This means that there are 5 actual words in the teacher's speech features that are identical to the semantic words. For the student's speech features, the semantic words that are identical to the actual words are: participate, discuss, viewpoint, and give an example. This means that there are 4 actual words in the student's speech features that are identical to the semantic words. Based on this example, the similarity of the interactive speech and behavior features between the teacher and the student can be calculated. In this case, the similarity of the interactive speech and behavior features between the teacher and the student can be 5 / 8=0.625, or 62.5%.

[0122] This similarity calculation method can evaluate the similarity of the interactive speech and behavior characteristics of teachers and students by comparing the matching degree between actual words and semantic words, and more accurately match the interactive voice characteristics of teachers and students.

[0123] In addition, this method has different activity judgment criteria for different class natures.

[0124] For example, there are three types of classes: the first is the communication between teachers and students before class, the second is the communication between teachers and students during class, and the third is the communication between teachers and students after class. Fig.13 , see the discussion section in class Fig.14 , see the after-class discussion section Fig.15 , Fig.13 It shows that there are more communication sessions before class. The teacher will release information and topics. Students need to preview the class based on the information released by the teacher and ask their own questions about the aspects they do not understand. Therefore, the period before class is a period with a relatively large demand for interaction. The pre-class atmosphere is ideal only when the activity level reaches more than 80% at this time. Fig.14 It shows that during the class, the teacher first checks the attendance, then organizes students to discuss during the lecture, and finally explains the released tests. The whole process is mainly based on the teacher's explanation, and students do not need to interact too much. At this time, the classroom activity reaches 50%, which is an ideal teaching environment. Fig.15 It shows that in the after-class communication session, the teacher mainly corrects the students' homework and answers their questions. The after-class communication process is also active and the classroom atmosphere is ideal when it reaches 80%.

[0125] The above mainly introduces the method of the embodiment of the present application in conjunction with the accompanying drawings. It should be noted that all the numerical values ​​appearing in the above are only used for examples and do not constitute specific limitations on the present application. It should also be understood that although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence, these steps are not necessarily performed in sequence in the order shown in the figure. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a part of the steps or stages in other steps. The system of the embodiment of the present application is introduced below in conjunction with the accompanying drawings. For the sake of brevity, when introducing the system below, it will be appropriately omitted, and the relevant content can refer to the relevant description in the above method introduction, and will not be repeated.

[0126] Fig.16 It is a structural diagram of a supervision system for learning multi-modal recognition teaching behaviors provided in an embodiment of the present application.

[0127] like Fig.16 As shown, the system 1600 includes a collection module 1601, a processing module 1602 and an activity determination module 1603. The system 1600 can execute any of the above-mentioned supervision methods for multi-modal recognition of teaching behaviors. For example, the collection module 1601 can be used to execute step S301, the processing module 1602 can be used to execute step S302, and the activity determination module 1603 can be used to execute step S303.

[0128] Fig.17 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig.17 As shown, the electronic device 3000 of this embodiment includes: at least one processor 3001 ( Fig.17 Only one is shown), a memory 3002, and a computer program 3003 stored in the memory 3002 and executable on at least one processor 3001, where the processor 3001 implements the steps in the above embodiments when executing the computer program 3003.

[0129] The processor 3001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] In some embodiments, the memory 3002 may be an internal storage unit of the electronic device 3000, such as a hard disk or memory of the electronic device 3000. In other embodiments, the memory 3002 may also be an external storage device of the electronic device 3000, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 3000. Further, the memory 3002 may also include both an internal storage unit of the electronic device 3000 and an external storage device. The memory 3002 is used to store an operating system, an application program, a boot loader (Boot Loader) data, and other programs, such as program codes of a computer program, etc. The memory 3002 may also be used to temporarily store data that has been output or is to be output.

[0131] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0132] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0133] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0134] The embodiments of the present application provide a computer program product, which can implement the above methods when the computer program product is run on a computer.

[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or system that can carry the computer program code to the camera system / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0136] It should be understood that the size of the sequence number of each step in the above-described embodiment does not mean the order of execution, and the order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application. In the description, for the purpose of illustration rather than for limitation, specific details such as specific system structure, technology and the like are proposed, so as to thoroughly understand the embodiment of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, systems, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application.

[0137] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0138] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0139] Additionally, in the specification of the present application and the description of the appended claims, the terms “include,” “comprise,” “have,” and their variations all mean “including but not limited to,” unless otherwise specifically emphasized.

[0140] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0141] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0142] In the embodiments provided in the present application, it should be understood that the disclosed systems, computer devices and methods can be implemented in other ways. For example, the system and computer device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0143] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A supervision method for multi-modal recognition of teaching behavior, characterized in that: include: Get the classroom media stream to be determined; Determining the teacher's speech and behavior characteristics and the student's speech and behavior characteristics according to the classroom media stream; Determine the level of classroom activity based on the characteristics of teachers' and students' words and deeds.

2. A method for supervising multi-modal recognition teaching behaviors as claimed in claim 1, characterized in that: The speech and behavior features include actions and voices; and determining the teacher's speech and behavior features and the student's speech and behavior features according to the classroom media stream includes: Extracting actions in the video and sounds in the audio of the media stream, wherein the actions include teacher actions and student actions, and the sounds include teacher voices and student voices; Determine the teacher's actions and voice based on the actions in the video and the sounds in the audio, and extract the teacher's speech and behavior features; Determine student actions and voices based on the actions in the video and the sounds in the audio, and extract student speech and behavior features.

3. A method for supervising multi-modal recognition teaching behaviors as claimed in claim 2, characterized in that: Determine the teacher's actions and voice according to the actions in the video and the voice in the audio, and extract the teacher's speech and behavior features; determine the student's actions and voice according to the actions in the video and the voice in the audio, and extract the student's speech and behavior features, including: According to a preset face information database, combined with the face information in the media stream, the position of the teacher and the position of each student are determined, and the human body movements in the area where each teacher is located and the human body movements in the area where each student is located are intercepted to obtain the behavioral characteristics of each teacher and each student; the human body movements include lip movement dynamics and torso movements; Determine the speech of each teacher based on the lip movement dynamics captured in the area where each teacher is located and the audio in the media stream to obtain the speech features of each teacher; According to the lip movement dynamics intercepted in the location area of ​​each student and the audio in the media stream, the voice uttered by each student is determined to obtain the voice features of each student.

4. A method for supervising multi-modal recognition teaching behaviors as claimed in claim 3, characterized in that: The method of determining the voice emitted by each teacher based on the lip movement dynamics intercepted in the location area of ​​each teacher and the audio in the media stream to obtain the voice features of each teacher, and determining the voice emitted by each student based on the lip movement dynamics intercepted in the location area of ​​each student and the audio in the media stream to obtain the voice features of each student includes: According to the preset lip movement dynamic library, the estimated speech corresponding to the lip movement dynamics of each teacher and each student is determined; Parsing the audio in the media stream to generate multiple segments of actual speech; Match each of the estimated voices with the actual voice, determine the actual voice corresponding to each lip movement, and use the actual voice corresponding to each lip movement as the voice feature of the corresponding person.

5. A method for supervising multi-modal recognition teaching behaviors as claimed in claim 1, characterized in that: Determining the classroom activity based on the teacher's speech and behavior characteristics and the student's speech and behavior characteristics includes: According to a preset interactive speech and behavior feature database between teachers and students, combining the speech and behavior features of each teacher and the speech and behavior features of each student, determining the interactive speech and behavior features of each teacher and all the speech and behavior features of each student; For each teacher and student, the activity level of each teacher and each student is determined based on the proportion of the corresponding interactive speech and behavior characteristics to the corresponding total speech and behavior characteristics.

6. A method for supervising multi-modal recognition teaching behaviors as claimed in claim 5, characterized in that: The interactive speech and behavior features of each teacher and each student are determined based on the preset interactive speech and behavior feature database between the teacher and the student, and the interactive speech and behavior features of all the speech and behavior features of each teacher and each student are determined, including: According to the preset database of interactive speech and behavior characteristics between teachers and students, the similarity between the interactive speech and behavior characteristics of each teacher and each student is determined by combining the speech and behavior characteristics of each teacher and the speech and behavior characteristics of each student; Determining whether the similarity of the interactive speech and behavior features is greater than a preset threshold; If the speech and behavior characteristics are judged to be interactive speech and behavior characteristics, otherwise they are teacher speech and behavior characteristics and student speech and behavior characteristics respectively.

7. A method for supervising multi-modal recognition of teaching behaviors as claimed in claim 6, wherein the similarity of speech and behavior features comprises: Posture similarity, characterized in that the similarity of the interactive speech and behavior features of each teacher and each student is determined based on a preset database of interactive speech and behavior features between teachers and students, combined with the speech and behavior features of each teacher and the speech and behavior features of each student, and includes: The joint position coordinates of each teacher and each student's actual posture are taken as the actual posture vector; The posture similarity is determined based on the distance between the actual posture vector and the posture vectors of each teacher and each student in the interactive speech and behavior feature database.

8. The method for supervising multi-modal recognition of teaching behaviors according to claim 6, wherein the similarity of speech and behavior features comprises: Semantic similarity, characterized in that the similarity of the interactive speech and behavior features of each teacher and each student is determined based on a preset database of interactive speech and behavior features between teachers and students, combined with the speech and behavior features of each teacher and the speech and behavior features of each student, and further includes: Parse the audio of each teacher and each student, determine the phonetic text, and extract the actual words in the phonetic text; The speech similarity is determined based on the number of identical actual words with the semantic words of each teacher and each student in the interactive speech and behavior database.

9. A supervision system for multi-modal recognition of teaching behavior, characterized in that: include: The acquisition module obtains the classroom media stream to be determined; A processing module, for determining the teacher's speech and behavior characteristics and the student's speech and behavior characteristics according to the classroom media stream; The activity determination module determines the classroom activity based on the characteristics of the teacher's words and deeds and the students' words and deeds.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images