A teaching interaction situation identification method, device, equipment and storage medium

Abstract

The present disclosure provides a teaching interaction situation identification method, device, equipment and storage medium, which are applied to a teaching scene provided with an audio acquisition device and a video acquisition device. The audio acquisition device and the video acquisition device are used to collect a teaching video of teaching activities in the teaching scene, and then the teacher's movements and the student's movements are analyzed from the teaching video, and the teacher's movements and the student's movements are integrated and processed to obtain the teacher-student interaction behavior. Based on the teacher-student interaction behavior, the teaching interaction situation can be effectively identified. Not only can the influence of the action behavior and speech behavior of each teacher and student be comprehensively identified to obtain the teacher-student interaction behavior, but also the teaching interaction situation corresponding to the teacher-student interaction behavior can be summarized based on the analysis of the teacher-student interaction behavior, thereby improving the detection of the types of teacher-student interaction behaviors and meeting the requirements for interaction behavior identification in complex teaching activities. The recognition accuracy is high and the recognition speed is fast, which greatly improves the ability to identify teaching interaction situations.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a teaching interaction situation identification method, device, equipment and storage medium. Background Art

[0002] With the development and progress of the times, strengthening the comprehensive quality education and ideological and moral education of young people has become an increasingly important issue, and the smart classroom teaching model has emerged. A smart classroom refers to a classroom learning environment composed of teaching media involving information technology applications. The introduction of information technology and terminal devices throughout the teaching process enables modern technology-driven teaching, making the classroom more intelligent and efficient.

[0003] In the existing teaching methods, the identification of teacher-student interaction in teaching activities is mainly carried out through manual observation and recording to evaluate and identify the interaction in teaching activities. The identification method has certain subjectivity and limitations, and it takes a long time to obtain the identification results, which consumes a lot of time and energy. It is difficult to meet the requirements for interaction identification in complex teaching activities. The recognition accuracy is low, it takes a long time, and the coverage of the identification of teacher-student interaction in teaching activities is not high. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a method, apparatus, device and storage medium for identifying teaching interaction situations.

[0005] The present disclosure provides a method for identifying teaching interaction situations, which is applied to a teaching scenario provided with an audio acquisition device and a video acquisition device. The method includes:

[0006] Acquire a teaching video of a teaching activity in the teaching scene, which is captured by the audio capture device and the video capture device;

[0007] Processing the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other;

[0008] Based on the teacher's action and the student's action, integrating to obtain the teacher-student interaction behavior;

[0009] Based on the teacher-student interaction behavior, the teaching interaction situation is determined.

[0010] In an optional implementation, the acquiring of a teaching video of a teaching activity in the teaching scene, which is captured by the audio capture device and the video capture device, includes:

[0011] Determining the audio acquisition device and the video acquisition device set in the teaching scene;

[0012] Acquiring audio data collected by the audio acquisition device and image data collected by the video acquisition device;

[0013] Performing screening and noise reduction processing on the collected audio data to obtain processed audio data;

[0014] Performing region division processing on the collected image data to obtain processed image data, wherein the image in the processed image data is divided into an information area where students are located and a teaching area where a teacher is located;

[0015] The processed image data and the processed audio data are integrated to obtain the teaching video.

[0016] In an optional implementation, the processing of the teaching video to obtain at least one teacher action and at least one student action includes:

[0017] Extracting a video image file and a video audio file from the teaching video;

[0018] Based on a preset time interval, the video image file is sliced ​​according to the preset time interval to obtain an image slice file;

[0019] For the video and audio files, obtaining an interactive audio file containing teacher-student interaction from the video and audio files;

[0020] Slice the interactive audio file to obtain an audio slice file;

[0021] Recognition processing is performed on the image segment file and the audio segment file respectively to obtain at least one teacher action and at least one student action.

[0022] In an optional embodiment, the performing recognition processing on the image segment file to obtain at least one teacher action and at least one student action includes:

[0023] For the image slice files, classifying the images of the image slice files to obtain teacher image slice files and student image slice files;

[0024] For the teacher image segment file, identifying that the teacher is in the information area, there is at least one person in the information area, the teacher is holding a smart mobile terminal, the teacher's head is turned at an angle of not less than 45 degrees, and the teacher is lowering his head;

[0025] With respect to the student image segmentation file, it is identified that there is at least one person in the information area and an action of the student holding the smart mobile terminal.

[0026] In an optional implementation, the integration of the teacher-student interaction behavior based on the teacher action and the student action includes:

[0027] Determining an interactive behavior of the teacher manipulating the technology based on the teacher being in the information area, at least one person being in the information area, the teacher holding a smart mobile terminal, the teacher turning his head at an angle of not less than 45 degrees, and the teacher lowering his head;

[0028] Based on the existence of at least one person in the information area and the action of the student holding the smart mobile terminal, the interactive behavior of the student manipulating the technology is determined.

[0029] In an optional implementation manner, determining the teaching interaction situation based on the teacher-student interaction behavior includes:

[0030] Based on the interactive behavior of the teacher manipulating the technology and the interactive behavior of the students manipulating the technology, it is determined that the teacher and the students use technology to interact in the teaching activity, and it is determined that the atmosphere between the teacher and the students in the teaching activity is harmonious.

[0031] In an optional implementation, the identifying and processing the audio segment file to obtain at least one teacher action includes:

[0032] For the audio segment file, extract at least one keyword from the audio segment file;

[0033] Based on a preset interaction scenario, determining whether the audio segment file matches the interaction scenario;

[0034] If there is a match, determining at least one of the associated sample words that matches the at least one keyword based on a plurality of associated sample words stored in a preset associated word library;

[0035] If there is no match, determining at least one public sample word that matches the at least one keyword based on a plurality of public sample words stored in a preset public vocabulary;

[0036] Based on the matched associated sample words and the common sample words, an action of obtaining the teacher's speech is determined.

[0037] In an optional implementation, the integration of the teacher-student interaction behavior based on the teacher action and the student action includes:

[0038] Based on the teacher's speaking actions, the teacher's interactive behaviors in the speech dimension are determined. The teacher's interactive behaviors in the speech dimension include one or more of the teacher receiving emotions, the teacher praising or encouraging, the teacher adopting students' opinions, the teacher asking questions, the teacher lecturing, the teacher instructing, and the teacher criticizing or maintaining the teacher's authority. The teacher's interactive behaviors in asking questions include asking open-ended questions and asking closed-ended questions.

[0039] In an optional embodiment, the method includes:

[0040] When the interactive behavior of the teacher asking questions is determined, locating the moment when the teacher asking questions occurs;

[0041] For the image segmentation file, determining whether the student's hand-raising action is recognized;

[0042] If the student does not raise his hand, it is determined that the student's interactive behavior of passive response is obtained;

[0043] If the student raises his hand, it is confirmed that the student actively speaks.

[0044] In an optional embodiment, the method includes:

[0045] In the case where it is determined that the student actively speaks in an interactive behavior, determining whether the location exists in the audio segment file corresponding to the student's active speaking interactive behavior;

[0046] If the positioning exists, determining that the student actively responds to the interactive behavior;

[0047] If the positioning does not exist, it is determined that an interactive behavior of the student actively asking questions is obtained.

[0048] In an optional implementation manner, determining the teaching interaction situation based on the teacher-student interaction behavior includes:

[0049] Based on the interactive behaviors directed at the teacher and the interactive behaviors directed at the students, the participation rates of the teacher and the students in the teaching activity, the guidance provided by the teacher to the students, and the feedback provided by the students to the teacher are determined.

[0050] In an optional embodiment, the method includes:

[0051] For the audio segment file, determining whether the audio is silent;

[0052] If the audio is silent, determine whether the student's head-lowering action is recognized based on the image segment file corresponding to the audio segment file;

[0053] If students lower their heads, ensure that they engage in quiet interaction that is beneficial to teaching;

[0054] If the audio is audible, determine whether the sound is noisy;

[0055] If so, determine whether the student is in a disorderly state;

[0056] If students are disruptive, be sure to get chaotic interactions that are not conducive to teaching.

[0057] If the students are not in a chaotic situation, determine whether they are grouped;

[0058] If so, make sure to get the interactive behavior of students discussing with their peers.

[0059] In an optional implementation manner, determining the teaching interaction situation based on the teacher-student interaction behavior includes:

[0060] Based on the quiet interactive behaviors that are beneficial to teaching, the chaotic interactive behaviors that are not helpful to teaching, and the interactive behaviors of students discussing with their peers, determining the ratio of students' orderly and chaotic situations in the teaching activities;

[0061] Based on the ratio, it is determined whether there is harmonious interaction or a gap between the teacher and the students in the teaching activity.

[0062] In an optional implementation, after integrating the teacher-student interaction behavior based on the teacher's action and the student's action, the method includes:

[0063] Processing the video image file and the video audio file corresponding to the teacher-student interaction behavior in chronological order to obtain processed new video image files and new video audio files;

[0064] For the processed new video image file and the new video audio file, determine whether there is duplication or missing;

[0065] If there are duplications or omissions, adjusting the new video image file and / or the new video audio file where duplications or omissions occur;

[0066] If there is no duplication or missing, determine the teacher-student interaction behavior corresponding to the new video image file and the new video audio file.

[0067] The present disclosure also provides a teaching interaction situation recognition device, which is applied to a teaching scenario provided with an audio acquisition device and a video acquisition device, and includes:

[0068] An acquisition module, configured to acquire a teaching video of the teaching activity in the teaching scene, which is acquired by the audio acquisition device and the video acquisition device;

[0069] a processing module, configured to process the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other;

[0070] An integration module, configured to integrate the teacher-student interaction behavior based on the teacher's actions and the student's actions;

[0071] The judgment module determines the teaching interaction situation based on the teacher-student interaction behavior.

[0072] In an optional implementation, the acquisition module is specifically configured to:

[0073] Determining the audio acquisition device and the video acquisition device set in the teaching scene;

[0074] Acquiring audio data collected by the audio acquisition device and image data collected by the video acquisition device;

[0075] Performing screening and noise reduction processing on the collected audio data to obtain processed audio data;

[0076] Performing region division processing on the collected image data to obtain processed image data, wherein the image in the processed image data is divided into an information area where students are located and a teaching area where a teacher is located;

[0077] The processed image data and the processed audio data are integrated to obtain the teaching video.

[0078] In an optional implementation manner, the processing module is specifically configured to:

[0079] Extracting a video image file and a video audio file from the teaching video;

[0080] Based on a preset time interval, the video image file is sliced ​​according to the preset time interval to obtain an image slice file;

[0081] For the video and audio files, obtaining an interactive audio file containing teacher-student interaction from the video and audio files;

[0082] Slice the interactive audio file to obtain an audio slice file;

[0083] Recognition processing is performed on the image segment file and the audio segment file respectively to obtain at least one teacher action and at least one student action.

[0084] In an optional embodiment, when the processing module is used to perform recognition processing on the image segment file to obtain at least one teacher action and at least one student action, it is specifically used to:

[0085] For the image slice files, classifying the images of the image slice files to obtain teacher image slice files and student image slice files;

[0086] For the teacher image segment file, identifying that the teacher is in the information area, there is at least one person in the information area, the teacher is holding a smart mobile terminal, the teacher's head is turned at an angle of not less than 45 degrees, and the teacher is lowering his head;

[0087] With respect to the student image segmentation file, it is identified that there is at least one person in the information area and an action of the student holding the smart mobile terminal.

[0088] In an optional implementation manner, the integration module is specifically configured to:

[0089] Determining an interactive behavior of the teacher manipulating the technology based on the teacher being in the information area, at least one person being in the information area, the teacher holding a smart mobile terminal, the teacher turning his head at an angle of not less than 45 degrees, and the teacher lowering his head;

[0090] Based on the existence of at least one person in the information area and the action of the student holding the smart mobile terminal, the interactive behavior of the student manipulating the technology is determined.

[0091] In an optional implementation manner, the judgment module is specifically configured to:

[0092] Based on the interactive behavior of the teacher manipulating the technology and the interactive behavior of the students manipulating the technology, it is determined that the teacher and the students use technology to interact in the teaching activity, and it is determined that the atmosphere between the teacher and the students in the teaching activity is harmonious.

[0093] In an optional embodiment, when the processing module is used to perform recognition processing on the audio segment file to obtain at least one teacher action, it is specifically used to:

[0094] For the audio segment file, extract at least one keyword from the audio segment file;

[0095] Based on a preset interaction scenario, determining whether the audio segment file matches the interaction scenario;

[0096] If there is a match, determining at least one of the associated sample words that matches the at least one keyword based on a plurality of associated sample words stored in a preset associated word library;

[0097] If there is no match, determining at least one public sample word that matches the at least one keyword based on a plurality of public sample words stored in a preset public vocabulary;

[0098] Based on the matched associated sample words and the common sample words, an action of obtaining the teacher's speech is determined.

[0099] In an optional implementation manner, the integration module is specifically configured to:

[0100] Based on the teacher's speaking actions, the teacher's interactive behaviors in the speech dimension are determined. The teacher's interactive behaviors in the speech dimension include one or more of the teacher receiving emotions, the teacher praising or encouraging, the teacher adopting students' opinions, the teacher asking questions, the teacher lecturing, the teacher instructing, and the teacher criticizing or maintaining the teacher's authority. The teacher's interactive behaviors in asking questions include asking open-ended questions and asking closed-ended questions.

[0101] In an optional implementation, the integration module is further configured to:

[0102] When the interactive behavior of the teacher asking questions is determined, locating the moment when the teacher asking questions occurs;

[0103] For the image segmentation file, determining whether the student's hand-raising action is recognized;

[0104] If the student does not raise his hand, it is determined that the student's interactive behavior of passive response is obtained;

[0105] If the student raises his hand, it is confirmed that the student actively speaks.

[0106] In an optional implementation, the integration module is further configured to:

[0107] In the case where it is determined that the student actively speaks in an interactive behavior, determining whether the location exists in the audio segment file corresponding to the student's active speaking interactive behavior;

[0108] If the positioning exists, determining that the student actively responds to the interactive behavior;

[0109] If the positioning does not exist, it is determined that an interactive behavior of the student actively asking questions is obtained.

[0110] In an optional implementation manner, the judgment module is specifically configured to:

[0111] Based on the interactive behaviors directed at the teacher and the interactive behaviors directed at the students, the participation rates of the teacher and the students in the teaching activity, the guidance provided by the teacher to the students, and the feedback provided by the students to the teacher are determined.

[0112] In an optional implementation manner, the integration module is specifically configured to:

[0113] For the audio segment file, determining whether the audio is silent;

[0114] If the audio is silent, determine whether the student's head-lowering action is recognized based on the image segment file corresponding to the audio segment file;

[0115] If students lower their heads, ensure that they engage in quiet interaction that is beneficial to teaching;

[0116] If the audio is audible, determine whether the sound is noisy;

[0117] If so, determine whether the student is in a disorderly state;

[0118] If students are disruptive, be sure to get chaotic interactions that are not conducive to teaching.

[0119] If the students are not in a chaotic situation, determine whether they are grouped;

[0120] If so, make sure to get the interactive behavior of students discussing with their peers.

[0121] In an optional implementation manner, the judgment module is specifically configured to:

[0122] Based on the quiet interactive behaviors that are beneficial to teaching, the chaotic interactive behaviors that are not helpful to teaching, and the interactive behaviors of students discussing with their peers, determining the ratio of students' orderly and chaotic situations in the teaching activities;

[0123] Based on the ratio, it is determined whether there is harmonious interaction or a gap between the teacher and the students in the teaching activity.

[0124] In an optional embodiment, the device further includes a detection module, wherein the detection module is configured to:

[0125] Processing the video image file and the video audio file corresponding to the teacher-student interaction behavior in chronological order to obtain processed new video image files and new video audio files;

[0126] For the processed new video image file and the new video audio file, determine whether there is duplication or missing;

[0127] If there are duplications or omissions, adjusting the new video image file and / or the new video audio file where duplications or omissions occur;

[0128] If there is no duplication or missing, determine the teacher-student interaction behavior corresponding to the new video image file and the new video audio file.

[0129] An embodiment of the present disclosure also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned teaching interaction situation identification method are performed.

[0130] An embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned teaching interaction situation identification method are executed.

[0131] The teaching interaction situation identification method, device, equipment and storage medium provided by the embodiments of the present disclosure are applied to a teaching scene provided with an audio acquisition device and a video acquisition device. The method, device, equipment and storage medium can obtain a teaching video of teaching activities in the teaching scene that is captured by the audio acquisition device and the video acquisition device; the teaching video is processed to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other; based on the teacher action and the student action, the teacher-student interaction behavior is integrated; based on the teacher-student interaction behavior, the teaching interaction situation is determined.

[0132] In this way, the audio acquisition equipment and the video acquisition equipment are used to collect the teaching video of the teaching activities in the teaching scene, and then the teacher's movements and student movements are analyzed from the teaching video, and the teacher's movements and student movements are integrated and processed to obtain the teacher-student interaction behavior. Based on the teacher-student interaction behavior, the teaching interaction situation can be effectively identified. By applying the teaching model of the smart classroom, information technology and terminal equipment are introduced into the traditional teaching process as a carrier for identifying and presenting the teaching interaction situation. Not only can the influence of the action and speech behavior of each teacher and student be comprehensively identified, but also the teaching interaction situation corresponding to the teacher-student interaction behavior can be analyzed and summarized based on the interaction behavior between teachers and students and the interaction between teachers and students and the smart learning environment. Accurate judgments are made on the specific behaviors of teachers and students in the teaching activities, providing more objective and quantitative analysis results for the identification of teacher-student interaction situations in teaching activities, improving the types of detection of teacher-student interaction behaviors, meeting the requirements for interactive behavior identification in complex teaching activities, with high recognition accuracy and fast recognition speed, greatly improving the ability to identify teaching interaction situations.

[0133] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0134] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0135] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure is shown;

[0136] Figure 2 A flow chart of a teaching interaction situation identification method provided by an embodiment of the present disclosure is shown;

[0137] Figure 3 A flowchart showing another teaching interaction situation identification method provided by an embodiment of the present disclosure is shown;

[0138] Figure 4 A flowchart showing a specific method for identifying teacher-student interaction behavior in the teaching interaction situation identification method provided by an embodiment of the present disclosure is shown;

[0139] Figure 5 A flowchart showing another specific method for identifying teacher-student interaction behavior in the teaching interaction situation identification method provided by an embodiment of the present disclosure is shown;

[0140] Figure 6 A flowchart showing another specific method for identifying teacher-student interaction behavior in the teaching interaction situation identification method provided by an embodiment of the present disclosure is shown;

[0141] Figure 7 One of the schematic diagrams of a teaching interaction situation identification device provided by an embodiment of the present disclosure is shown;

[0142] Figure 8 A second schematic diagram of a teaching interaction situation identification device provided by an embodiment of the present disclosure is shown;

[0143] Figure 9 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0144] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0145] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0146] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0147] Research has found that in existing teaching methods, the identification of teacher-student interaction in teaching activities is mainly through manual observation and recording. For example, in the form of open classes, auditors manually observe and summarize during teaching activities. This identification method has certain subjectivity and limitations, and it takes a long time to obtain the identification results, which consumes a lot of time and energy. It is difficult to meet the requirements for identifying interaction situations in complex teaching activities. The recognition accuracy is low, it takes a long time, and the coverage of the identification of teacher-student interaction in teaching activities is not high.

[0148] Based on the above research, the present disclosure provides a method for identifying teaching interaction situations, which is applied to a teaching scene equipped with an audio acquisition device and a video acquisition device. The method can obtain a teaching video of teaching activities conducted in the teaching scene, which is captured by the audio acquisition device and the video acquisition device; process the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other; based on the teacher action and the student action, integrate the teacher-student interaction behavior; and determine the teaching interaction situation based on the teacher-student interaction behavior.

[0149] In this way, the audio acquisition equipment and the video acquisition equipment are used to collect the teaching video of the teaching activities in the teaching scene, and then the teacher's movements and student movements are analyzed from the teaching video, and the teacher's movements and student movements are integrated and processed to obtain the teacher-student interaction behavior. Based on the teacher-student interaction behavior, the teaching interaction situation can be effectively identified. By applying the teaching model of the smart classroom, information technology and terminal equipment are introduced into the traditional teaching process as a carrier for identifying and presenting the teaching interaction situation. Not only can the influence of the action and speech behavior of each teacher and student be comprehensively identified, but also the teaching interaction situation corresponding to the teacher-student interaction behavior can be analyzed and summarized based on the interaction behavior between teachers and students and the interaction between teachers and students and the smart learning environment. Accurate judgments are made on the specific behaviors of teachers and students in the teaching activities, providing more objective and quantitative analysis results for the identification of teacher-student interaction situations in teaching activities, improving the types of detection of teacher-student interaction behaviors, meeting the requirements for interactive behavior identification in complex teaching activities, with high recognition accuracy and fast recognition speed, greatly improving the ability to identify teaching interaction situations.

[0150] To facilitate understanding of this embodiment, we first provide a detailed introduction to a method for identifying interactive teaching situations disclosed in this embodiment. The method is typically executed by an electronic device with sufficient computing power, such as a terminal device, a server, or other processing device. In some possible implementations, this method can be implemented by a processor invoking computer-readable instructions stored in a memory.

[0151] See also Figure 1 , Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 As shown in , in teaching activities, the main body of interaction is students, and the interacting entities include teachers, teaching content, learning environment, and interaction between students. Among them, teaching content mainly includes teaching materials and resources. Teaching materials can include paper teaching materials and electronic teaching materials. Resources can include text, pictures, audio and video, etc. In addition, the learning environment mainly includes information technology and terminal equipment. Information technology mainly involves the Internet, big data, the Internet of Things, and artificial intelligence, etc. Terminal equipment can include mobile phones, tablets, electronic whiteboards, educational robots, etc.

[0152] During teaching activities, teachers and students can provide feedback, ask questions and answer questions, students can cooperate and discuss with each other, teachers can design teaching content, and teaching content can be provided for students to study and watch on their own, teachers can create situations for the learning environment, and the learning environment can be provided for students to operate and experiment. In addition, the teaching content requires the learning environment as a carrier for presenting knowledge and resources, and the learning environment can also provide corresponding functions to support the development of teaching.

[0153] See also Figure 2 , Figure 2 This is a flow chart of a teaching interaction situation identification method provided by the embodiment of the present disclosure. Figure 2 As shown, the teaching interaction situation identification method provided by the embodiment of the present disclosure includes:

[0154] S201: Acquire a teaching video of a teaching activity in the teaching scene, which is captured by the audio capture device and the video capture device.

[0155] In this step, when it is necessary to identify the teaching interaction situation of the teaching activities carried out in the teaching scene to determine the teaching interaction situation in the teaching activities, the audio acquisition device and the video acquisition device set in the teaching scene can be used to obtain the teaching video of the teaching activities carried out in the teaching scene captured by the audio acquisition device and the video acquisition device, so that the teaching video can be subsequently analyzed to obtain the teaching interaction situation.

[0156] It can be understood that the audio acquisition device and the video acquisition device are provided in the teaching scene, so the teaching video can be obtained with the help of the audio acquisition device and the video acquisition device.

[0157] Optionally, the audio capture device and the video capture device may be pre-installed in the teaching scene. For example, when it is necessary to identify the teaching interaction of teaching activities conducted in the teaching scene to determine the teaching interaction in the teaching activities, the teaching scene may be a multimedia classroom in which the audio capture device and the video capture device are pre-installed. In this way, the teaching video can be captured using the audio capture device and the video capture device pre-installed in the multimedia classroom.

[0158] Optionally, when it is necessary to record the teaching activities conducted in the teaching scene, the audio capture device and the video capture device may be set up before recording. For example, when it is necessary to identify the teaching interaction situation of the teaching activities conducted in the teaching scene to determine the teaching interaction situation in the teaching activities, the teaching scene may be an ordinary classroom without the audio capture device and the video capture device pre-installed. In this case, the audio capture device and the video capture device may be temporarily set up to achieve the capture of the teaching video.

[0159] Among them, teachers and students can wear the audio acquisition device on their bodies, and the audio acquisition device can be a microphone or a headset. In addition, the audio acquisition device can be fixed on a podium, desk or wall, and the audio acquisition device can also be a boom microphone or a lifting microphone. There is no limitation here.

[0160] Among them, the video acquisition device can be a fixed camera installed in different areas of the teaching scene, or a camera installed in a fixed area of ​​the teaching scene that can swing according to a preset angle, and no limitation is made here.

[0161] S202: Processing the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other.

[0162] In this step, when the teaching video is collected, at least one teacher action and at least one student action can be obtained from the teaching video through analysis and processing.

[0163] It can be understood that the teacher action can be an action identified with the teacher as the action subject, the student action can be an action identified with the student as the action subject, and the teacher action and the student action can be part of a set of continuous actions. Therefore, the teacher action and the student action are in a corresponding relationship with each other.

[0164] For example, the teacher's action of asking a question and the student's action of raising a hand can be connected into a set of continuous actions, and the two actions correspond to each other.

[0165] In the case of determining the teaching video, in order to parse and obtain the teacher action and the student action, in some possible implementations, processing the teaching video to obtain at least one teacher action and at least one student action includes:

[0166] Extracting a video image file and a video audio file from the teaching video;

[0167] Based on a preset time interval, the video image file is sliced ​​according to the preset time interval to obtain an image slice file;

[0168] For the video and audio files, obtaining an interactive audio file containing teacher-student interaction from the video and audio files;

[0169] Slice the interactive audio file to obtain an audio slice file;

[0170] Recognition processing is performed on the image segment file and the audio segment file respectively to obtain at least one teacher action and at least one student action.

[0171] In this step, when the teaching video is determined, it can be understood that the teaching video is the video to be analyzed. In order to analyze the teaching video, the image and audio of the teaching video can be extracted respectively to obtain the video image file and the video audio file extracted from the teaching video. Then, for the video image file, the video image file can be sliced ​​according to a preset time interval to obtain the image slice file after slice processing. For the video audio file, the video audio file can be matched with the interaction scenario based on a preset interaction scenario to obtain an interactive audio file that meets the interaction scenario. The interactive audio file can then be sliced ​​to obtain the audio slice file after slice processing. Then, the image slice file and the audio slice file can be identified and processed respectively to obtain at least one teacher action and at least one student action.

[0172] It should be noted that the specific duration of the preset time interval can be determined based on actual conditions, subject to the accurate identification and parsing of the image slice files after slicing, and is not limited herein. For example, if the preset time interval is three seconds, the video image file is sliced ​​every three seconds to obtain the image slice files.

[0173] S203: Based on the teacher's actions and the student's actions, integrate and obtain the teacher-student interaction behavior.

[0174] In this step, when the teacher's action and the student's action are determined, the teacher's action and the student's action that are related are integrated to obtain the teacher-student interaction behavior.

[0175] For example, when a teacher action of asking a question and a student action of raising a hand are obtained, an interactive behavior of the student actively responding can be integrated based on the teacher action of asking a question and the student action of raising a hand.

[0176] Here, the teacher-student interactive behavior includes interactive behavior directed at teachers and interactive behavior directed at students. The interactive behavior directed at teachers may include teacher action behavior and teacher speech behavior, and the interactive behavior directed at students may include student action behavior and student speech behavior.

[0177] S204: Determine the teaching interaction situation based on the teacher-student interaction behavior.

[0178] In this step, when the teacher-student interaction behavior is obtained, the teaching interaction situation in the teaching activity can be determined based on the interaction situation between the teacher and the students in the teacher-student interaction behavior.

[0179] Furthermore, after obtaining the teaching interaction situation of the teaching activities in the teaching scenario, teaching interaction situation feedback information can be generated and fed back to the teaching teacher so that the teacher can make adaptive adjustments to the teaching methods and teaching progress according to the teaching interaction situation.

[0180] The teaching interaction situation identification method provided by the embodiment of the present disclosure is applied to a teaching scene provided with an audio acquisition device and a video acquisition device. The method can obtain a teaching video of teaching activities in the teaching scene that is captured by the audio acquisition device and the video acquisition device; process the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other; based on the teacher action and the student action, the teacher-student interaction behavior is integrated; based on the teacher-student interaction behavior, the teaching interaction situation is determined.

[0181] In this way, the audio acquisition equipment and the video acquisition equipment are used to capture the teaching video of the teaching activities in the teaching scene, and then the teacher's movements and the student's movements are analyzed from the teaching video, and the teacher's movements and the student's movements are integrated and processed to obtain the teacher-student interaction behavior. Based on the teacher-student interaction behavior, the teaching interaction situation can be effectively identified. Not only can the influence of the action and speech behavior of each teacher and student be comprehensively identified, but the teaching interaction situation corresponding to the teacher-student interaction behavior can also be summarized based on the analysis of the teacher-student interaction behavior, so as to improve the detection of the types of teacher-student interaction behaviors and meet the requirements for interactive behavior identification in complex teaching activities. The recognition accuracy is high and the recognition speed is fast, which greatly improves the ability to identify teaching interaction situations.

[0182] See also Figure 3 , Figure 3 This is a flow chart of another teaching interaction situation identification method provided by the embodiment of the present disclosure. Figure 3 As shown, the teaching interaction situation identification method provided by the embodiment of the present disclosure includes:

[0183] S301: Acquire a teaching video of a teaching activity in the teaching scene, which is captured by the audio capture device and the video capture device.

[0184] S302: Processing the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other.

[0185] S303: Based on the teacher's actions and the student's actions, integrate and obtain the teacher-student interaction behavior.

[0186] S304: Processing the video image file and the video audio file corresponding to the teacher-student interaction behavior in chronological order to obtain processed new video image files and new video audio files.

[0187] In this step, when the teacher-student interaction behavior is obtained, the video image file and video audio file corresponding to the teacher-student interaction behavior can be determined first, and then the video image file and video audio file can be sorted according to the chronological order of the occurrence of the teacher-student interaction behavior in the teaching activity to obtain the processed new video image file and new video audio file.

[0188] In the actual processing process, each teacher-student interaction behavior can correspond to an interaction behavior coding data. When the teacher-student interaction behavior is obtained, the interaction behavior coding data corresponding to the teacher-student interaction behavior can be read, and then the interaction behavior coding data can be sorted in chronological order to obtain the sorted interaction behavior coding data and its corresponding new video image file and new video audio file.

[0189] S305: Determine whether there is any duplication or missing in the processed new video image file and the new video audio file.

[0190] In this step, when the processed new video image file and the new video audio file are obtained, it can be determined whether the new video image file and the new video audio file are repeated or missing.

[0191] In the actual processing process, the sorted interactive behavior coding data can be divided into nodes according to preset time intervals to obtain divided coding data nodes, and then it can be determined whether there is overlap or missing based on the coding data nodes.

[0192] S306: If there are duplications or omissions, adjust the new video image file and / or the new video audio file where duplications or omissions occur.

[0193] In this step, if there are duplications or omissions, the locations where the duplications or omissions occur can be located, and the new video image file and / or the new video audio file where the duplications or omissions occur can be adjusted.

[0194] In the actual processing process, if there are duplications or omissions, the overlapping or missing nodes can be located and adjusted using the new video image file and / or the new video audio file.

[0195] S307: If there is no duplication or missing, determine the teacher-student interaction behavior corresponding to the new video image file and the new video audio file.

[0196] In this step, if there is no duplication or missing situation, it can be understood that the teacher-student interaction behavior corresponding to the new video image file and the new video audio file at this time is the same as the teacher-student interaction behavior corresponding to the video image file and the video audio file before adjustment in chronological order. Therefore, the teacher-student interaction behavior corresponding to the new video image file and the new video audio file can be determined, so as to determine the teaching interaction situation based on the teacher-student interaction behavior corresponding to the new video image file and the new video audio file in subsequent processing.

[0197] S308: Determine the teaching interaction situation based on the teacher-student interaction behavior.

[0198] Among them, the description of steps S301 to S303 and step S308 can refer to the description of steps S201 to S204, and can achieve the same technical effects and solve the same technical problems, so they are not repeated here.

[0199] The above solution will be described below in conjunction with specific implementation methods.

[0200] In order to obtain the teaching video, in some possible implementations, obtaining the teaching video of the teaching activity in the teaching scene, which is captured by the audio capture device and the video capture device, includes:

[0201] Determining the audio acquisition device and the video acquisition device set in the teaching scene;

[0202] Acquiring audio data collected by the audio acquisition device and image data collected by the video acquisition device;

[0203] Performing screening and noise reduction processing on the collected audio data to obtain processed audio data;

[0204] Performing region division processing on the collected image data to obtain processed image data, wherein the image in the processed image data is divided into an information area where students are located and a teaching area where a teacher is located;

[0205] The processed image data and the processed audio data are integrated to obtain the teaching video.

[0206] In this step, in order to obtain the teaching video, the audio acquisition device and the video acquisition device set in the teaching scene can be first determined. It can be understood that the audio acquisition device can collect audio data of teaching activities in the teaching scene, and the video acquisition device can collect image data of teaching activities in the teaching scene. Therefore, the audio data collected by the audio acquisition device and the image data collected by the video acquisition device can be obtained. Here, in order to obtain the audio data and the image data of higher quality and convenient for subsequent recognition and processing, for the audio data, invalid audio can be screened and removed, and the filtered audio data can be subjected to noise reduction processing. For the image data, the image in the image data can be divided into regions to obtain the information area where the students are located and the teaching area where the teacher is located. Then, the processed image data and the processed audio data can be integrated to obtain the teaching video.

[0207] Optionally, the image data may include a panoramic image of the teacher and a panoramic image of the students, or may include a close-up image of the teacher and a close-up image of the students, which is not limited here.

[0208] Furthermore, in response to the administrator's operation, the teaching video can be modified or deleted, and the administrator can also view the processing status and details of the teaching video.

[0209] In some possible implementations, the performing recognition processing on the image segment file to obtain at least one teacher action and at least one student action includes:

[0210] For the image slice files, classifying the images of the image slice files to obtain teacher image slice files and student image slice files;

[0211] For the teacher image segment file, identifying that the teacher is in the information area, there is at least one person in the information area, the teacher is holding a smart mobile terminal, the teacher's head is turned at an angle of not less than 45 degrees, and the teacher is lowering his head;

[0212] With respect to the student image segmentation file, it is identified that there is at least one person in the information area and an action of the student holding the smart mobile terminal.

[0213] In this step, when the image slice file is obtained, the screen of the image slice file can be extracted, and the image slice file can be classified according to whether the screen displays a teacher or a student, so that a teacher image slice file and a student image slice file can be obtained. For the teacher image file, it can be identified that the teacher is in the information area, there is at least one person in the information area, the teacher holds a smart mobile terminal, the teacher turns his head at an angle of not less than forty-five degrees, and the teacher lowers his head. For the student image slice file, it can be identified that there is at least one person in the information area and the student holds a smart mobile terminal.

[0214] Here, it should be noted that before carrying out the teaching activities, the teachers and students in the teaching scene can be equipped with the smart mobile terminals in advance. For students, the smart mobile terminals can include learning tools such as mobile phones and tablets. For teachers, the smart mobile terminals can include teaching tools such as mobile phones, tablets, electronic whiteboards, and educational robots.

[0215] In some possible implementations, the teacher-student interaction behavior obtained by integrating the teacher action and the student action includes:

[0216] Determining an interactive behavior of the teacher manipulating the technology based on the teacher being in the information area, at least one person being in the information area, the teacher holding a smart mobile terminal, the teacher turning his head at an angle of not less than 45 degrees, and the teacher lowering his head;

[0217] Based on the existence of at least one person in the information area and the action of the student holding the smart mobile terminal, the interactive behavior of the student manipulating the technology is determined.

[0218] Here, you can also refer to Figure 4 , Figure 4 In the teaching interaction situation identification method provided in the embodiment of the present disclosure, a flowchart of a specific method for identifying teacher-student interaction behavior is provided. Figure 4 As shown in , when it is identified that the teacher is in the information area, there is at least one person in the information area, the teacher holds a smart mobile terminal, the teacher turns his head at an angle of not less than forty-five degrees, and the teacher lowers his head, it can be determined that the teacher's interactive behavior of manipulating the technology is obtained. When it is identified that there is at least one person in the information area and the student holds a smart mobile terminal, it can be determined that the student's interactive behavior of manipulating the technology is obtained.

[0219] In some possible implementations, determining the teaching interaction situation based on the teacher-student interaction behavior includes:

[0220] Based on the interactive behavior of the teacher manipulating the technology and the interactive behavior of the students manipulating the technology, it is determined that the teacher and the students use technology to interact in the teaching activity, and it is determined that the atmosphere between the teacher and the students in the teaching activity is harmonious.

[0221] In this step, after obtaining the teacher's interactive behavior using the technology and the student's interactive behavior using the technology, it can be determined that the teacher and student are interacting using the technology in the teaching activity. Furthermore, the frequency of the teacher and student interacting using the technology can be determined. If the frequency meets a preset threshold, it can be determined that the atmosphere between the teacher and student in the teaching activity is harmonious.

[0222] Furthermore, based on the frequency of the teacher and students using technology to interact, if the frequency does not meet the preset threshold, it is determined that there is a communication gap between the teacher and students in the teaching activity.

[0223] In some possible implementations, the performing recognition processing on the audio segment file to obtain at least one teacher action includes:

[0224] For the audio segment file, extract at least one keyword from the audio segment file;

[0225] Based on a preset interaction scenario, determining whether the audio segment file matches the interaction scenario;

[0226] If there is a match, determining at least one of the associated sample words that matches the at least one keyword based on a plurality of associated sample words stored in a preset associated word library;

[0227] If there is no match, determining at least one public sample word that matches the at least one keyword based on a plurality of public sample words stored in a preset public vocabulary;

[0228] Based on the matched associated sample words and the common sample words, an action of obtaining the teacher's speech is determined.

[0229] In this step, when the audio segment file is obtained, at least one keyword contained in the audio segment file can be extracted, and based on a preset interaction scenario, it is determined whether the audio segment file matches the interaction scenario, that is, whether the at least one keyword extracted from the audio segment file matches the interaction scenario. If it matches, based on a preset associated vocabulary, where a plurality of associated sample words are stored in the associated vocabulary, the at least one keyword is matched with the plurality of associated sample words to obtain at least one associated sample word that matches the at least one keyword; if it does not match, based on a preset public vocabulary, where a plurality of public sample words are stored in the public vocabulary, the at least one keyword is matched with the plurality of public sample words to obtain at least one public sample word that matches the at least one keyword, so that the action of the teacher speaking can be determined based on the matched associated sample words and the public sample words.

[0230] Here, the preset interaction context may be a context related to the teaching activity.

[0231] For example, when the teaching activity is a mathematics class, the preset interaction context is an interaction context in which a teacher teaches mathematics to students.

[0232] In some possible implementations, the teacher-student interaction behavior obtained by integrating the teacher action and the student action includes:

[0233] Based on the teacher's speaking actions, the teacher's interactive behaviors in the speech dimension are determined. The teacher's interactive behaviors in the speech dimension include one or more of the teacher receiving emotions, the teacher praising or encouraging, the teacher adopting students' opinions, the teacher asking questions, the teacher lecturing, the teacher instructing, and the teacher criticizing or maintaining the teacher's authority. The teacher's interactive behaviors in asking questions include asking open-ended questions and asking closed-ended questions.

[0234] In this step, when the teacher's speaking action is determined, the interactive behavior of the teacher's speech dimension can be determined based on the associated sample words and the common sample words matched by the audio segment file.

[0235] Furthermore, after obtaining the teacher's language expression through audio segmentation file recognition, the image segmentation file can be combined to jointly analyze and obtain the teacher's interactive behavior in the speech dimension, further ensuring the accuracy and completeness of behavior recognition.

[0236] Furthermore, based on the teacher's interactive behavior in the verbal dimension, the teacher's classroom teaching status can be determined, such as whether the teacher uses guiding language to guide students, whether the teacher lacks guidance for students, whether the teacher actively guides students' answers, or whether the teacher passively guides students' answers.

[0237] In some possible implementations, the method includes:

[0238] When the interactive behavior of the teacher asking questions is determined, locating the moment when the teacher asking questions occurs;

[0239] For the image segmentation file, determining whether the student's hand-raising action is recognized;

[0240] If the student does not raise his hand, it is determined that the student's interactive behavior of passive response is obtained;

[0241] If the student raises his hand, it is confirmed that the student actively speaks.

[0242] Here, you can also refer to Figure 5 , Figure 5 This is a flowchart of another specific method for identifying teacher-student interaction behavior in the teaching interaction situation identification method provided by the embodiment of the present disclosure, such as Figure 5 As shown in , when it is determined that an interactive behavior of a teacher asking a question is obtained, the moment when the teacher's question occurs can be located, so as to judge whether the student raises his hand for the image of the image segmentation file. If so, it is determined that an interactive behavior of a student passively responding is obtained; if not, it is determined that an interactive behavior of a student actively speaking is obtained.

[0243] In some possible implementations, the method includes:

[0244] In the case where it is determined that the student actively speaks in an interactive behavior, determining whether the location exists in the audio segment file corresponding to the student's active speaking interactive behavior;

[0245] If the positioning exists, determining that the student actively responds to the interactive behavior;

[0246] If the positioning does not exist, it is determined that an interactive behavior of the student actively asking questions is obtained.

[0247] Here, in the case of determining that the student's interactive behavior of actively speaking is obtained, it can be further determined whether the positioning exists in the audio segment file corresponding to the interactive behavior of the student's active speaking. It can be understood that if the positioning exists, it means that the student's hand-raising action is a response to the teacher's question, thereby determining that the student's interactive behavior of actively answering is obtained. If the positioning does not exist, it means that the student's hand-raising action is a response to the teacher's question, thereby determining that the student's interactive behavior of actively asking questions is obtained.

[0248] In some possible implementations, determining the teaching interaction situation based on the teacher-student interaction behavior includes:

[0249] Based on the interactive behaviors directed at the teacher and the interactive behaviors directed at the students, the participation rates of the teacher and the students in the teaching activity, the guidance provided by the teacher to the students, and the feedback provided by the students to the teacher are determined.

[0250] In this step, by determining the interactive behaviors directed at the teacher and the interactive behaviors directed at the students, the participation rates of the teacher and the students in the teaching activities can be determined respectively. Furthermore, the guidance of the teacher to the students can be obtained based on the interactive behaviors directed at the teacher, and the feedback of the students to the teacher can be obtained based on the interactive behaviors directed at the students.

[0251] Optionally, pie charts, bar charts and other charts can be used to present the results of the teaching situation analysis from the aspects of teacher speech ratio, teacher behavior ratio, student speech ratio, student behavior ratio, etc.

[0252] In some possible implementations, the method includes:

[0253] For the audio segment file, determining whether the audio is silent;

[0254] If the audio is silent, determine whether the student's head-lowering action is recognized based on the image segment file corresponding to the audio segment file;

[0255] If students lower their heads, ensure that they engage in quiet interaction that is beneficial to teaching;

[0256] If the audio is audible, determine whether the sound is noisy;

[0257] If so, determine whether the student is in a disorderly state;

[0258] If students are disruptive, be sure to get chaotic interactions that are not conducive to teaching.

[0259] If the students are not in a chaotic situation, determine whether they are grouped;

[0260] If so, make sure to get the interactive behavior of students discussing with their peers.

[0261] Here, you can also refer to Figure 6 , Figure 6 In the teaching interaction situation identification method provided in the embodiment of the present disclosure, there is also a flowchart of a specific method for identifying teacher-student interaction behavior, such as Figure 6As shown in , the audio segment file is read to determine whether the audio is quiet. If the audio is quiet, the image segment file corresponding to the audio segment file is used to determine whether the student is looking down. If the student is looking down, it is determined that a quiet interactive behavior beneficial to teaching is obtained. Here, it can be understood that the student looking down and being quiet means that the student may be in a state of studying or writing, so the silence at this time is beneficial to teaching activities.

[0262] Furthermore, if the audio is audible, determine whether the sound is noisy. If the sound is noisy, determine whether the students are in a state of confusion. If the students are in a state of confusion, it can be understood that the noisy sound and the students' confusion are not conducive to the development of teaching activities, thereby determining that the chaotic interactive behavior that is not conducive to teaching is obtained. If the students are not in a state of confusion, determine whether the students are in groups. If so, it can be understood that the noisy sound is caused by the students' group discussions, thereby determining that the students are in an interactive behavior of discussing with their peers.

[0263] In some possible implementations, determining the teaching interaction situation based on the teacher-student interaction behavior includes:

[0264] Based on the quiet interactive behaviors that are beneficial to teaching, the chaotic interactive behaviors that are not helpful to teaching, and the interactive behaviors of students discussing with their peers, determining the ratio of students' orderly and chaotic situations in the teaching activities;

[0265] Based on the ratio, it is determined whether there is harmonious interaction or a gap between the teacher and the students in the teaching activity.

[0266] In this step, when determining the silent interactive behaviors that are beneficial to teaching, the chaotic interactive behaviors that are not helpful to teaching, and the interactive behaviors of students discussing with their peers, the orderly and chaotic situations of students in the teaching activities can be obtained, and thus the ratio of the orderly and chaotic situations can be obtained. Based on the preset ratio, the ratio for the teaching activity is compared with the preset ratio, and then it is determined whether the interaction between teachers and students in the teaching activities is harmonious or there is a gap.

[0267] Furthermore, for each of the teaching activities, a classroom report can be generated for each of the teaching activities. The classroom report can convert and present the teaching interaction situation into data visualization. The classroom report can include a classroom timeline, a classroom transition matrix and a classroom behavior ratio. The classroom timeline can be used to describe the changes in the interaction between teachers and students in the teaching activity, the classroom transition matrix can be used to describe the teaching structure in the teaching activity, and the classroom behavior analysis ratio can be used to describe the overall structure of the teaching activity.

[0268] Furthermore, classroom reports for the same class, the same teacher, the same subject, the same grade, etc. can be integrated into class reports, teacher reports, subject reports, grade reports, etc., so that teachers can intuitively understand the teaching situation from different angles.

[0269] The teaching interaction situation identification method provided by the embodiment of the present disclosure is applied to a teaching scene provided with an audio acquisition device and a video acquisition device. The method can obtain a teaching video of teaching activities in the teaching scene that is captured by the audio acquisition device and the video acquisition device; process the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other; based on the teacher action and the student action, the teacher-student interaction behavior is integrated; based on the teacher-student interaction behavior, the teaching interaction situation is determined.

[0270] In this way, the audio acquisition equipment and the video acquisition equipment are used to collect the teaching video of the teaching activities in the teaching scene, and then the teacher's movements and the student's movements are analyzed from the teaching video, and the teacher's movements and the student's movements are integrated and processed to obtain the teacher-student interaction behavior. Based on the teacher-student interaction behavior, the teaching interaction situation can be effectively identified. By applying the teaching model of the smart classroom, information technology and terminal equipment are introduced into the traditional teaching process as a carrier for identifying and presenting the teaching interaction situation. Not only can the influence of the action and speech behavior of each teacher and student be comprehensively identified, but also the teaching interaction situation corresponding to the teacher-student interaction behavior can be analyzed and summarized based on the interaction behavior between teachers and students and the interaction between teachers and students and the smart learning environment. Accurate judgments are made on the specific behaviors of teachers and students in teaching activities, providing more objective and quantitative analysis results for the identification of teacher-student interaction situations in teaching activities, meeting the requirements for interactive behavior identification in complex teaching activities, with high recognition accuracy and fast recognition speed, and improving the ability to identify teaching interaction situations.

[0271] Those skilled in the art will understand that in the above method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0272] Based on the same inventive concept, the embodiment of the present disclosure also provides a teaching interaction situation identification device corresponding to the teaching interaction situation identification method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned teaching interaction situation identification method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0273] See also Figure 7 and Figure 8 , Figure 7This is one of the schematic diagrams of a teaching interaction situation identification device provided by an embodiment of the present disclosure. Figure 8 This is a second schematic diagram of a teaching interaction situation identification device provided by an embodiment of the present disclosure. Figure 7 As shown in , the teaching interaction situation identification device 700 provided by the embodiment of the present disclosure includes:

[0274] The acquisition module 710 is used to acquire the teaching video of the teaching activity in the teaching scene, which is acquired by the audio acquisition device and the video acquisition device.

[0275] The processing module 720 is used to process the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other.

[0276] The integration module 730 is used to integrate the teacher-student interaction behavior based on the teacher's action and the student's action.

[0277] The judgment module 740 determines the teaching interaction situation based on the teacher-student interaction behavior.

[0278] In an optional implementation, the acquisition module 710 is specifically configured to:

[0279] Determining the audio acquisition device and the video acquisition device set in the teaching scene;

[0280] Acquiring audio data collected by the audio acquisition device and image data collected by the video acquisition device;

[0281] Performing screening and noise reduction processing on the collected audio data to obtain processed audio data;

[0282] Performing region division processing on the collected image data to obtain processed image data, wherein the image in the processed image data is divided into an information area where students are located and a teaching area where a teacher is located;

[0283] The processed image data and the processed audio data are integrated to obtain the teaching video.

[0284] In an optional implementation, the processing module 720 is specifically configured to:

[0285] Extracting a video image file and a video audio file from the teaching video;

[0286] Based on a preset time interval, the video image file is sliced ​​according to the preset time interval to obtain an image slice file;

[0287] For the video and audio files, obtaining an interactive audio file containing teacher-student interaction from the video and audio files;

[0288] Slice the interactive audio file to obtain an audio slice file;

[0289] Recognition processing is performed on the image segment file and the audio segment file respectively to obtain at least one teacher action and at least one student action.

[0290] In an optional embodiment, when the processing module 720 is used to perform recognition processing on the image segment file to obtain at least one teacher action and at least one student action, it is specifically used to:

[0291] For the image slice files, classifying the images of the image slice files to obtain teacher image slice files and student image slice files;

[0292] For the teacher image segment file, identifying that the teacher is in the information area, there is at least one person in the information area, the teacher is holding a smart mobile terminal, the teacher's head is turned at an angle of not less than 45 degrees, and the teacher is lowering his head;

[0293] With respect to the student image segmentation file, it is identified that there is at least one person in the information area and an action of the student holding the smart mobile terminal.

[0294] In an optional implementation, the integration module 730 is specifically configured to:

[0295] Determining an interactive behavior of the teacher manipulating the technology based on the teacher being in the information area, at least one person being in the information area, the teacher holding a smart mobile terminal, the teacher turning his head at an angle of not less than 45 degrees, and the teacher lowering his head;

[0296] Based on the existence of at least one person in the information area and the action of the student holding the smart mobile terminal, the interactive behavior of the student manipulating the technology is determined.

[0297] In an optional implementation manner, the judgment module 740 is specifically configured to:

[0298] Based on the interactive behavior of the teacher manipulating the technology and the interactive behavior of the students manipulating the technology, it is determined that the teacher and the students use technology to interact in the teaching activity, and it is determined that the atmosphere between the teacher and the students in the teaching activity is harmonious.

[0299] In an optional implementation, when the processing module 720 is used to perform recognition processing on the audio segment file and obtain at least one teacher action, it is specifically used to:

[0300] For the audio segment file, extract at least one keyword from the audio segment file;

[0301] Based on a preset interaction scenario, determining whether the audio segment file matches the interaction scenario;

[0302] If there is a match, determining at least one of the associated sample words that matches the at least one keyword based on a plurality of associated sample words stored in a preset associated word library;

[0303] If there is no match, determining at least one public sample word that matches the at least one keyword based on a plurality of public sample words stored in a preset public vocabulary;

[0304] Based on the matched associated sample words and the common sample words, an action of obtaining the teacher's speech is determined.

[0305] In an optional implementation, the integration module 730 is specifically configured to:

[0306] Based on the teacher's speaking actions, the teacher's interactive behaviors in the speech dimension are determined. The teacher's interactive behaviors in the speech dimension include one or more of the teacher receiving emotions, the teacher praising or encouraging, the teacher adopting students' opinions, the teacher asking questions, the teacher lecturing, the teacher instructing, and the teacher criticizing or maintaining the teacher's authority. The teacher's interactive behaviors in asking questions include asking open-ended questions and asking closed-ended questions.

[0307] In an optional implementation, the integration module 730 is further configured to:

[0308] When the interactive behavior of the teacher asking questions is determined, locating the moment when the teacher asking questions occurs;

[0309] For the image segmentation file, determining whether the student's hand-raising action is recognized;

[0310] If the student does not raise his hand, it is determined that the student's interactive behavior of passive response is obtained;

[0311] If the student raises his hand, it is confirmed that the student actively speaks.

[0312] In an optional implementation, the integration module 730 is further configured to:

[0313] In the case where it is determined that the student actively speaks in an interactive behavior, determining whether the location exists in the audio segment file corresponding to the student's active speaking interactive behavior;

[0314] If the positioning exists, determining that the student actively responds to the interactive behavior;

[0315] If the positioning does not exist, it is determined that an interactive behavior of the student actively asking questions is obtained.

[0316] In an optional implementation manner, the judgment module 740 is specifically configured to:

[0317] Based on the interactive behaviors directed at the teacher and the interactive behaviors directed at the students, the participation rates of the teacher and the students in the teaching activity, the guidance provided by the teacher to the students, and the feedback provided by the students to the teacher are determined.

[0318] In an optional implementation, the integration module 730 is specifically configured to:

[0319] For the audio segment file, determining whether the audio is silent;

[0320] If the audio is silent, determine whether the student's head-lowering action is recognized based on the image segment file corresponding to the audio segment file;

[0321] If students lower their heads, ensure that they engage in quiet interaction that is beneficial to teaching;

[0322] If the audio is audible, determine whether the sound is noisy;

[0323] If so, determine whether the student is in a disorderly state;

[0324] If students are disruptive, be sure to get chaotic interactions that are not conducive to teaching.

[0325] If the students are not in a chaotic situation, determine whether they are grouped;

[0326] If so, make sure to get the interactive behavior of students discussing with their peers.

[0327] In an optional implementation manner, the judgment module 740 is specifically configured to:

[0328] Based on the quiet interactive behaviors that are beneficial to teaching, the chaotic interactive behaviors that are not helpful to teaching, and the interactive behaviors of students discussing with their peers, determining the ratio of students' orderly and chaotic situations in the teaching activities;

[0329] Based on the ratio, it is determined whether there is harmonious interaction or a gap between the teacher and the students in the teaching activity.

[0330] In an optional embodiment, as Figure 8 As shown, the apparatus further includes a detection module 750, and the detection module 750 is configured to:

[0331] Processing the video image file and the video audio file corresponding to the teacher-student interaction behavior in chronological order to obtain processed new video image files and new video audio files;

[0332] For the processed new video image file and the new video audio file, determine whether there is duplication or missing;

[0333] If there are duplications or omissions, adjusting the new video image file and / or the new video audio file where duplications or omissions occur;

[0334] If there is no duplication or missing, determine the teacher-student interaction behavior corresponding to the new video image file and the new video audio file.

[0335] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0336] The teaching interaction situation identification device provided by the embodiment of the present disclosure is applied to a teaching scene provided with an audio acquisition device and a video acquisition device. It can obtain a teaching video of teaching activities in the teaching scene that is captured by the audio acquisition device and the video acquisition device; process the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other; based on the teacher action and the student action, integrate the teacher-student interaction behavior; and determine the teaching interaction situation based on the teacher-student interaction behavior.

[0337] In this way, the audio acquisition equipment and the video acquisition equipment are used to collect the teaching video of the teaching activities in the teaching scene, and then the teacher's movements and student movements are analyzed from the teaching video, and the teacher's movements and student movements are integrated and processed to obtain the teacher-student interaction behavior. Based on the teacher-student interaction behavior, the teaching interaction situation can be effectively identified. By applying the teaching model of the smart classroom, information technology and terminal equipment are introduced into the traditional teaching process as a carrier for identifying and presenting the teaching interaction situation. Not only can the influence of the action and speech behavior of each teacher and student be comprehensively identified, but also the teaching interaction situation corresponding to the teacher-student interaction behavior can be analyzed and summarized based on the interaction behavior between teachers and students and the interaction between teachers and students and the smart learning environment. Accurate judgments are made on the specific behaviors of teachers and students in the teaching activities, providing more objective and quantitative analysis results for the identification of teacher-student interaction situations in teaching activities, improving the types of detection of teacher-student interaction behaviors, meeting the requirements for interactive behavior identification in complex teaching activities, with high recognition accuracy and fast recognition speed, greatly improving the ability to identify teaching interaction situations.

[0338] Corresponding to Figure 2 and Figure 3 The teaching interaction situation recognition method in the present disclosure embodiment also provides an electronic device 900, such as Figure 9 FIG. 1 is a schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present disclosure, including:

[0339] Processor 910, memory 920, and bus 930; memory 920 is used to store execution instructions, including internal memory 921 and external memory 922; the memory 921 here is also called internal memory, which is used to temporarily store the calculation data in the processor 910, as well as the data exchanged with the external memory 922 such as the hard disk. The processor 910 exchanges data with the external memory 922 through the memory 921. When the electronic device 900 is running, the processor 910 and the memory 920 communicate through the bus 930, so that the processor 910 can execute the steps of the above-mentioned teaching interaction situation identification method.

[0340] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the teaching interaction situation identification method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0341] The embodiments of the present disclosure also provide a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, the steps of the teaching interaction situation identification method described in the above method embodiment can be executed. For details, please refer to the above method embodiment and will not be repeated here.

[0342] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0343] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0344] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0345] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0346] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0347] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A method for identifying teaching interaction situations, characterized in that: Applied to a teaching scenario provided with an audio acquisition device and a video acquisition device, the method includes: Acquire a teaching video of a teaching activity in the teaching scene, which is captured by the audio capture device and the video capture device; Processing the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other; Based on the teacher's action and the student's action, integrating to obtain the teacher-student interaction behavior; Determining the teaching interaction situation based on the teacher-student interaction behavior; The processing of the teaching video to obtain at least one teacher action and at least one student action includes: Extracting a video image file and a video audio file from the teaching video; Based on a preset time interval, the video image file is sliced ​​according to the preset time interval to obtain an image slice file; For the video and audio files, obtaining an interactive audio file containing teacher-student interaction from the video and audio files; Slice the interactive audio file to obtain an audio slice file; Performing recognition processing on the image segment file and the audio segment file respectively to obtain at least one teacher action and at least one student action; The audio segment file is identified and processed to obtain at least one teacher action, including: For the audio segment file, extract at least one keyword from the audio segment file; Based on a preset interaction scenario, determining whether the audio segment file matches the interaction scenario; If there is a match, determining at least one of the associated sample words that matches the at least one keyword based on a plurality of associated sample words stored in a preset associated word library; If there is no match, determining at least one public sample word that matches the at least one keyword based on a plurality of public sample words stored in a preset public vocabulary; Based on the matched associated sample words and the common sample words, an action of obtaining the teacher's speech is determined.

2. The method according to claim 1, characterized in that The obtaining of the teaching video of the teaching activity in the teaching scene, which is collected by the audio collection device and the video collection device, includes: Determining the audio acquisition device and the video acquisition device set in the teaching scene; Acquiring audio data collected by the audio acquisition device and image data collected by the video acquisition device; Performing screening and noise reduction processing on the collected audio data to obtain processed audio data; Performing region division processing on the collected image data to obtain processed image data, wherein the image in the processed image data is divided into an information area where students are located and a teaching area where a teacher is located; The processed image data and the processed audio data are integrated to obtain the teaching video.

3. The method according to claim 1, characterized in that The performing recognition processing on the image segment file to obtain at least one teacher action and at least one student action includes: For the image slice files, classifying the images of the image slice files to obtain teacher image slice files and student image slice files; For the teacher image segment file, identifying that the teacher is in the information area, there is at least one person in the information area, the teacher is holding a smart mobile terminal, the teacher's head is turned at an angle of not less than 45 degrees, and the teacher is lowering his head; With respect to the student image segmentation file, it is identified that there is at least one person in the information area and an action of the student holding the smart mobile terminal.

4. The method according to claim 3, characterized in that The teacher-student interaction behavior obtained by integrating the teacher action and the student action includes: Determining an interactive behavior of the teacher manipulating the technology based on the teacher being in the information area, at least one person being in the information area, the teacher holding a smart mobile terminal, the teacher turning his head at an angle of not less than 45 degrees, and the teacher lowering his head; Based on the existence of at least one person in the information area and the action of the student holding the smart mobile terminal, the interactive behavior of the student manipulating the technology is determined.

5. The method according to claim 4, characterized in that Determining the teaching interaction situation based on the teacher-student interaction behavior includes: Based on the interactive behavior of the teacher manipulating the technology and the interactive behavior of the students manipulating the technology, it is determined that the teacher and the students use technology to interact in the teaching activity, and it is determined that the atmosphere between the teacher and the students in the teaching activity is harmonious.

6. The method according to claim 1, characterized in that The teacher-student interaction behavior obtained by integrating the teacher action and the student action includes: Based on the teacher's speaking actions, the teacher's interactive behaviors in the speech dimension are determined. The teacher's interactive behaviors in the speech dimension include one or more of the teacher receiving emotions, the teacher praising or encouraging, the teacher adopting students' opinions, the teacher asking questions, the teacher lecturing, the teacher instructing, and the teacher criticizing or maintaining the teacher's authority. The teacher's interactive behaviors in asking questions include asking open-ended questions and asking closed-ended questions.

7. The method according to claim 6, characterized in that The method comprises: When the interactive behavior of the teacher asking questions is determined, locating the moment when the teacher asking questions occurs; For the image segmentation file, determining whether the student's hand-raising action is recognized; If the student does not raise his hand, it is determined that the student's interactive behavior of passive response is obtained; If the student raises his hand, it is confirmed that the student actively speaks.

8. The method according to claim 7, characterized in that The method comprises: In the case where it is determined that the student actively speaks in an interactive behavior, determining whether the location exists in the audio segment file corresponding to the student's active speaking interactive behavior; If the positioning exists, determining that the student actively responds to the interactive behavior; If the positioning does not exist, it is determined that an interactive behavior of the student actively asking questions is obtained.

9. The method according to any one of claims 6 to 8, characterized in that: Determining the teaching interaction situation based on the teacher-student interaction behavior includes: Based on the interactive behaviors directed at the teacher and the interactive behaviors directed at the students, the participation rates of the teacher and the students in the teaching activity, the guidance provided by the teacher to the students, and the feedback provided by the students to the teacher are determined.

10. The method according to claim 1, characterized in that The method comprises: For the audio segment file, determining whether the audio is silent; If the audio is silent, determine whether the student's head-lowering action is recognized based on the image segment file corresponding to the audio segment file; If students lower their heads, ensure that they engage in quiet interaction that is beneficial to teaching; If the audio is audible, determine whether the sound is noisy; If so, determine whether the student is in a disorderly state; If students are disruptive, ensure that the disruptive interactions are not conducive to teaching; If the students are not in a chaotic situation, determine whether they are grouped; If so, make sure to get the interactive behavior of students discussing with their peers.

11. The method according to claim 10, characterized in that Determining the teaching interaction situation based on the teacher-student interaction behavior includes: Based on the quiet interactive behaviors that are beneficial to teaching, the chaotic interactive behaviors that are not helpful to teaching, and the interactive behaviors of students discussing with their peers, determining the ratio of students' orderly and chaotic situations in the teaching activities; Based on the ratio, it is determined whether there is harmonious interaction or a gap between the teacher and the students in the teaching activity.

12. The method according to claim 1, characterized in that After integrating the teacher-student interaction behavior based on the teacher's action and the student's action, the method includes: Processing the video image file and the video audio file corresponding to the teacher-student interaction behavior in chronological order to obtain processed new video image files and new video audio files; For the processed new video image file and the new video audio file, determine whether there is duplication or missing; If there are duplications or omissions, adjusting the new video image file and / or the new video audio file where duplications or omissions occur; If there is no duplication or missing, determine the teacher-student interaction behavior corresponding to the new video image file and the new video audio file.

13. A teaching interaction situation recognition device, characterized in that: Applicable to a teaching scenario provided with an audio acquisition device and a video acquisition device, the device comprises: An acquisition module, configured to acquire a teaching video of the teaching activity in the teaching scene, which is acquired by the audio acquisition device and the video acquisition device; a processing module, configured to process the teaching video to obtain at least one teacher action and at least one student action, wherein the teacher action and the student action are in a corresponding relationship with each other; An integration module, configured to integrate the teacher-student interaction behavior based on the teacher's actions and the student's actions; A judgment module, which determines the teaching interaction situation based on the teacher-student interaction behavior; The processing module is specifically used for: Extracting a video image file and a video audio file from the teaching video; Based on a preset time interval, the video image file is sliced ​​according to the preset time interval to obtain an image slice file; For the video and audio files, obtaining an interactive audio file containing teacher-student interaction from the video and audio files; Slice the interactive audio file to obtain an audio slice file; Performing recognition processing on the image segment file and the audio segment file respectively to obtain at least one teacher action and at least one student action; When the processing module is used to perform recognition processing on the audio segment file and obtain at least one teacher action, it is specifically used to: For the audio segment file, extract at least one keyword from the audio segment file; Based on a preset interaction scenario, determining whether the audio segment file matches the interaction scenario; If there is a match, determining at least one of the associated sample words that matches the at least one keyword based on a plurality of associated sample words stored in a preset associated word library; If there is no match, determining at least one public sample word that matches the at least one keyword based on a plurality of public sample words stored in a preset public vocabulary; Based on the matched associated sample words and the common sample words, an action of obtaining the teacher's speech is determined.

14. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the teaching interaction situation identification method as described in any one of claims 1 to 12 are performed.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the teaching interaction situation identification method according to any one of claims 1 to 12.

Images