Teaching behavior analysis method and device, equipment and storage medium

Abstract

The invention provides a teaching behavior analysis method and device, equipment and a storage medium, relates to the technical field of audio and video processing, and solves the problems that classroom teaching contents and teaching behaviors are difficult to associate in analysis of teaching audios and videos, and analysis of teaching quality is not reasonable enough. According to the scheme, corresponding knowledge point information, teaching behaviors of teachers and learning behaviors of students can be determined according to analysis results of visual analysis and semantic analysis of teaching audios and videos, classroom teaching contents and teaching behaviors are better associated, and cognition of a user on a teaching effect and improvement of teaching quality are facilitated.

Description

Technical Field

[0001] The present application relates to the field of audio and video processing technology, and in particular to a teaching behavior analysis method, device, equipment and storage medium. Background Art

[0002] Digital and media-based intelligent information technology is changing the traditional teaching model, and the application of educational informatization in the field of classroom teaching is becoming more and more extensive. For example, in order to collect and record the behavior of teachers and students in the classroom, multimedia equipment such as cameras, video recorders, and microphones are usually set up in the classroom to record teaching audio and video for teaching behavior analysis.

[0003] In related technologies, some manufacturers provide services that only identify classroom behaviors through visual analysis and voice analysis of videos, and count the corresponding duration to determine the duration and proportion of classroom behaviors.

[0004] However, the inventors found in actual applications that the above data have a weak correlation with teaching quality and cannot be correlated with classroom teaching content and teacher teaching behavior. The solutions provided by related technologies cannot provide users with more objective and reasonable teaching quality analysis, which is not conducive to users' understanding of teaching effectiveness and improvement of teaching quality. The solutions provided by related technologies are not practical enough. Summary of the invention

[0005] The present application provides a teaching behavior analysis method, device, equipment and storage medium, which solves the problem that it is difficult to associate classroom teaching content and teaching behavior in the analysis of teaching audio and video, and the analysis of teaching quality is not reasonable enough. This solution can better associate classroom teaching content and teaching behavior, and obtain more reasonable teaching quality analysis results.

[0006] In a first aspect, the present application provides a teaching behavior analysis method, which is applied to a terminal device, wherein the terminal device is communicatively connected to an audio and video acquisition component in a classroom scene to receive teaching audio and video collected by the audio and video acquisition component, and the method comprises:

[0007] Obtain the course information of the current classroom, and perform semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain the knowledge point information related to the classroom teaching content;

[0008] Determine the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student based on the results of the visual analysis and semantic analysis of the teaching audio and video;

[0009] Based on the timeline of the teaching audio and video, determine the time nodes corresponding to each knowledge point information and each teaching behavior, and generate a teaching process timing diagram;

[0010] According to the timeline of teaching audio and video and learning behavior, determine the learning engagement of students at each sampling moment, and associate the learning engagement with the corresponding knowledge point information and the time nodes of each teaching behavior;

[0011] A corresponding display window is generated and displayed, and the display window displays teaching audio and video, learning engagement and a teaching process timing diagram.

[0012] In a second aspect, the present application further provides a teaching behavior analysis device, the device comprising:

[0013] A knowledge point extraction module is configured to obtain the course information of the current classroom, and perform semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain the knowledge point information related to the classroom teaching content;

[0014] A behavior analysis module, configured to determine the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student based on the analysis results of the visual analysis and semantic analysis of the teaching audio and video;

[0015] An image generation module is configured to determine the time nodes corresponding to each knowledge point information and each teaching behavior based on the time axis of the teaching audio and video, and generate a teaching process timing diagram;

[0016] A data association module is configured to determine the learning engagement of students at each sampling moment according to the timeline of the teaching audio and video and the learning behavior, and associate the learning engagement with the corresponding knowledge point information and the time nodes of each teaching behavior;

[0017] The window generation module is configured to generate and display a corresponding display window, which displays teaching audio and video, learning engagement and a teaching process timing diagram.

[0018] In a third aspect, the present application further provides a terminal device, including:

[0019] one or more processors;

[0020] The storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the teaching behavior analysis method provided above.

[0021] In a fourth aspect, the present application also provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to execute the teaching behavior analysis method provided above.

[0022] This application solution determines the knowledge point information and corresponding teaching behavior of the current class in the audio data and video screen through visual analysis and semantic analysis of teaching audio and video, and also distinguishes the teaching behavior of the teacher and the learning behavior of the students, so as to associate the classroom teaching content with the teaching behavior of the teacher, and also determines the corresponding learning engagement corresponding to the learning behavior of the students, and then associates the classroom teaching content again. The solution can use the association between classroom teaching content and teaching behavior to objectively and reasonably reflect the teaching quality, so that the device can generate output results that can reflect the teaching quality, which is conducive to the user's cognition of teaching effects and the improvement of teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of the steps of the teaching behavior analysis method provided in one embodiment of the present application;

[0024] Figure 2 A flowchart of the steps for obtaining knowledge point information provided in an embodiment of the present application;

[0025] Figure 3 A flowchart of the steps for determining teaching behavior provided in an embodiment of the present application;

[0026] Figure 4 A flowchart of the steps for determining a learning behavior provided in an embodiment of the present application;

[0027] Figure 5 A flowchart of the steps for determining teaching behavior provided in one embodiment of the present application;

[0028] Figure 6 A schematic diagram of a teaching process timing diagram provided in an embodiment of the present application;

[0029] Figure 7 A flowchart of the steps for determining a student's learning engagement provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of a display window provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of a teaching behavior analysis device provided in one embodiment of the present application;

[0032] Fig.10 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only the parts related to the embodiments of the present application rather than all structures are shown in the accompanying drawings, and those skilled in the art should be able to think of it after reading the specification of this application that as long as the technical features do not contradict each other, any combination of the technical features can constitute an optional implementation method.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship. In the description of the present application, "multiple" means two or more, and "several" means one or more.

[0035] In the teaching model of educational informatization, the recorded teaching audio and video are usually carried out in places such as multimedia classrooms that are equipped with audio and video acquisition components such as cameras and microphones. For example, a fixed-position camera is used to capture the image of the teacher teaching in the classroom, another camera is used to capture the image of the students, and a microphone is used to capture the audio of the teaching process to form teaching audio and video.

[0036] The present application scheme provides a teaching behavior analysis method, which is applied to a terminal device, and the terminal device is communicatively connected with the above-mentioned audio and video acquisition component to receive corresponding teaching audio and video.

[0037] Figure 1 A flowchart of the steps of the teaching behavior analysis method provided in one embodiment of the present application is shown in FIG. Figure 1 As shown, the teaching behavior analysis method provided in this application includes the following steps:

[0038] Step S110: Obtain the course information of the current classroom, and perform semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain knowledge point information related to the classroom teaching content.

[0039] For the teaching mode of educational informatization, classrooms are usually equipped with large-screen display devices such as smart blackboards and interactive smart tablets as teaching equipment to display courseware and materials related to the current course to achieve auxiliary teaching. Correspondingly, the corresponding teaching equipment in each classroom can report course information such as the current course, opened courseware, etc. at the corresponding operation nodes through pre-set embedding points.

[0040] The terminal device can perform semantic analysis on the received teaching audio and video in combination with the course information. It can be understood that when performing semantic analysis, the device extracts audio data from the teaching audio and video, and then locates the corresponding audio clips to extract keywords, tone and other information, and then obtains knowledge point information related to the current classroom teaching content.

[0041] It can be imagined that the knowledge point information associated with the current classroom teaching content can be recorded in the form of a knowledge graph to record the knowledge points corresponding to the current courseware, so that after determining the courseware to be displayed, the terminal device can determine the knowledge points covered in this lesson, and then through semantic analysis of the audio data, determine the knowledge point information involved in the video at the corresponding moment.

[0042] Step S120: Determine the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student based on the results of the visual analysis and semantic analysis of the teaching audio and video.

[0043] The terminal device will also analyze the actions of the characters (such as teachers and students) that appear in the teaching audio and video, that is, visual analysis of the teaching audio and video. Of course, if it is necessary to determine the specific behaviors of the teacher and the student, it is also necessary to combine the semantic analysis of the teaching audio and video, that is, the terminal device needs to determine the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student based on the analysis results of the visual analysis and semantic analysis of the teaching audio and video.

[0044] It should be noted that teaching behavior refers to the behavior taken by teachers and students in the teaching process to achieve certain teaching goals. In this regard, teaching behavior in this application includes the teaching behavior corresponding to teachers and the learning behavior corresponding to students.

[0045] For example, the teacher's teaching behavior includes asking questions. To determine this behavior, the terminal device needs to perform visual analysis and semantic analysis on the teaching audio and video, and finally determine that the teacher's current teaching behavior is a questioning behavior. For example, the terminal device determines through visual analysis that the teacher is currently doing oral actions, and accordingly, performs semantic analysis on the audio clips during the time period when the teacher is doing oral actions. When it is determined that the teacher has issued an inquiry statement, the terminal device determines that the teacher's teaching behavior is a questioning behavior.

[0046] Step S130: Based on the time axis of the teaching audio and video, determine the time nodes corresponding to each knowledge point information and each teaching behavior, and generate a teaching process timing diagram.

[0047] After determining the knowledge point information involved in the teaching audio and video, the terminal device needs to arrange the knowledge point information according to the time axis of the teaching audio and video, that is, according to the time sequence, and at the same time, the terminal device will also locate the teaching behavior on the time axis accordingly. In addition, there are multiple time nodes on the time axis, and the time nodes on the time axis correspond to the knowledge point information and the teaching behavior. For example, at a moment, the teacher is currently explaining a knowledge point, that is, the moment corresponds to a time node, and the time node also corresponds to the knowledge point information, and the time node also corresponds to the teacher's teaching behavior.

[0048] The terminal device generates a corresponding teaching process timing diagram in combination with the time node, knowledge point information and teaching behavior for teaching behavior analysis. It should be noted that in some embodiments, the time node corresponding to the knowledge point information also corresponds to a teaching behavior of the teacher.

[0049] Step S140: Determine the student's learning engagement at each sampling moment based on the timeline of the teaching audio and video and the learning behavior, and associate the learning engagement with the corresponding knowledge point information and the time nodes of each teaching behavior.

[0050] The terminal device also needs to record the student's learning engagement, and determine the student's learning engagement according to the time sequence of the teaching audio and video and the student's learning behavior. Specifically, the student's learning engagement at the sampling moment can refer to the student's head-up rate and / or hand-raising rate to characterize the student's learning engagement.

[0051] It should be noted that the above-mentioned sampling moment can be a moment determined by a preset sampling interval; the sampling moment can also correspond to the moment of a time node, that is, the terminal device can detect the learning engagement at the corresponding moment of each time node, and then combine the learning engagement at each sampling moment to generate a learning engagement curve for the current class. Therefore, the learning engagement can be associated with the time node.

[0052] Step S150, generate and display a corresponding display window, which displays teaching audio and video, learning engagement and a teaching process timing diagram.

[0053] The display window is used to deliver relevant information to the user, and displays the video screen of the teaching audio and video, the learning engagement of the students, and the teaching process timing diagram. Therefore, users such as teachers can use the display window to review the teaching behaviors they adopted for different knowledge points in the teaching process in this class, as well as the impact of the corresponding teaching behaviors and knowledge points on the learning engagement of students.

[0054] From the above scheme, it can be seen that the terminal device can determine the knowledge point information and corresponding teaching behavior of the current class in the audio data and video screen through visual analysis and semantic analysis of the teaching audio and video, and also distinguish the teaching behavior of the teacher and the learning behavior of the students to associate the classroom teaching content with the teaching behavior of the teacher, and also determine the corresponding learning engagement corresponding to the learning behavior of the students, and then associate the classroom teaching content again. The scheme can use the association between classroom teaching content and teaching behavior to objectively and reasonably reflect the teaching quality, so that the device can generate output results that can reflect the teaching quality, which is conducive to the user's cognition of teaching effects and the improvement of teaching quality.

[0055] In some embodiments, for obtaining knowledge point information associated with classroom teaching content, the terminal device can perform semantic analysis on the teaching audio and video in combination with the course information to obtain the knowledge point information. It is conceivable that in addition to displaying courseware, the teaching equipment in the classroom also records the corresponding course information, and the course information includes the teaching subject and the corresponding subject chapter corresponding to the courseware currently being used.

[0056] like Figure 2 As shown, Figure 2 The flowchart of the steps for obtaining knowledge point information provided in an embodiment of the present application, the specific steps are as follows:

[0057] Step S210: Obtain course information from the teaching device.

[0058] Step S220: Based on the audio data of the teaching audio and video, identify the speaker to determine the identity of the speaker and the audio segment corresponding to the speaker.

[0059] Step S230: When the speaker is a teacher, based on the knowledge point content corresponding to the subject chapter, keywords are extracted from the audio clip in which the speaker is the teacher, and knowledge point information is determined based on the keywords.

[0060] It is understandable that after obtaining the course information, the terminal device can obtain the knowledge points associated with the current classroom teaching content. Moreover, the terminal device extracts the audio clip corresponding to the teacher in the teaching audio and video. It is conceivable that the teacher and the students may interact throughout the class. Correspondingly, there will be multiple speakers in the audio data, but there is only one speaker corresponding to the teacher identity. Therefore, the terminal device can extract the audio clip whose speaker is the teacher.

[0061] For example, the terminal device can identify the speaker through voiceprint recognition technology, and accordingly, the terminal device can also divide the audio data into multiple audio segments according to the identity and number of speakers. Of course, in order to determine the knowledge point information, the terminal device needs to extract the audio segment corresponding to the speaker being the teacher, thereby extracting the keywords in the audio segment related to the knowledge point content corresponding to the subject chapter, and then determining the knowledge point information.

[0062] For example, the teacher explained in class that "Ampere force is the force exerted on a current-carrying wire in a magnetic field." For this audio clip, the terminal device can extract the keyword "Ampere force" and further determine that the current knowledge point information is related to Ampere force.

[0063] Therefore, the terminal device identifies the speaker and extracts keywords from the audio clip where the speaker is the teacher, so as to accurately determine the current knowledge point information, which is conducive to better linking classroom teaching content and teaching behavior.

[0064] In one embodiment, when the terminal device extracts keywords from the audio clip of the teacher's speech, it obtains multiple different keywords. At this time, the terminal device needs to filter the keywords in combination with the temporal order of the knowledge point content to lock the target keyword.

[0065] It is understandable that there is a sequence in the arrangement of knowledge point contents. That is, in order to facilitate students' understanding, teachers will explain the contents of each knowledge point in a corresponding logical order during the teaching process. Therefore, in the teaching audio and video, the contents of each knowledge point appear in chronological order.

[0066] For example, taking the above sentence "Ampere force is the force exerted on a current-carrying wire in a magnetic field" as an example, in order to let students understand the Ampere force more clearly, generally, teachers will introduce the concept of magnetic field, that is, first explain the knowledge point of magnetic field, and then explain the knowledge point of Ampere force, thereby forming different teaching emphases in time sequence.

[0067] Therefore, the keywords extracted by the terminal device from the audio clip are "Ampere force" and "magnetic field". From the above, it can be seen that the keyword "magnetic field" should have been mentioned in the previous sequence, so it is filtered and the target keyword is locked as "Ampere force".

[0068] Therefore, the terminal device can lock the target keyword from multiple keywords by combining the temporal order of the knowledge point content, and thus determine the knowledge point information more accurately and effectively.

[0069] In some embodiments, the terminal device needs to determine the teaching behavior of the teacher and the learning behavior of the student based on the results of the visual analysis and semantic analysis of the teaching audio and video, such as Figure 3 As shown, Figure 3 A flowchart of steps for determining teaching behavior provided in an embodiment of the present application, the specific steps are as follows:

[0070] Step S310: Perform target detection on the captured video images to locate the teachers and students in the video images.

[0071] Step S320: Based on a preset posture recognition algorithm, determine the student action and the teacher action, and determine a first action moment of the student action and a second action moment of the teacher action.

[0072] Step S330: Determine the student action as a learning behavior based on the semantic analysis result of the first audio segment.

[0073] Step S340: Determine that the teacher's action is a teaching behavior in combination with the semantic analysis result of the second audio segment.

[0074] It is understandable that the video images in the teaching audio and video are collected to locate the teachers and students appearing in the video images through the target detection algorithm. In addition, the student movements and teacher movements are determined by the posture recognition algorithm, such as determining the key points of the human body and determining the posture of the target based on the key points of the human body, and then determining the student movements and teacher movements.

[0075] Of course, when determining the student's action and the teacher's action, the terminal device also records the corresponding action moments, such as the first action moment corresponding to the student's action and the second target moment corresponding to the teacher's action; and then determines the corresponding audio clips to determine the student's and teacher's actions. The first audio clip is an audio clip corresponding to the target duration of the first action moment, and the second audio clip is an audio clip corresponding to the target duration of the second action moment.

[0076] For the first audio segment and the second audio segment, the terminal performs semantic analysis on them respectively through a semantic analysis algorithm to obtain corresponding semantic analysis results, thereby determining whether the action currently performed by the student is a learning behavior and whether the action currently performed by the teacher is a teaching behavior.

[0077] It can be imagined that, taking student actions as an example, such as raising hands, in a complex classroom environment, students may not raise their hands in response to the teacher's questions. That is, students may raise their hands for other reasons, such as raising their hands to indicate to the teacher that they need to leave the classroom temporarily.

[0078] Therefore, for the actions of students and teachers in teaching audio and video, the terminal device determines whether it belongs to teaching behavior by combining the analysis results of visual analysis and semantic analysis, so as to more accurately determine the teacher's teaching behavior and the student's learning behavior, which helps to better associate classroom teaching content and teaching behavior.

[0079] It should be noted that the above-mentioned object detection algorithm, pose recognition algorithm, and semantic analysis algorithm can all adopt algorithms in related technologies, such as the Two-Stage algorithm, the algorithm based on the Stacked Hourglass Model (SHM), etc.

[0080] Figure 4 The following is a flowchart of the steps for determining learning behavior provided by an embodiment of the present application. As shown in the figure, in one embodiment, when the student's action is a raising hand action, the terminal device needs to analyze the corresponding audio segment to determine whether the raising hand action is a learning behavior. The specific steps are as follows:

[0081] Step S410: In the case where the student's action is a raising hand action, based on the action moment corresponding to the raising hand action, select the first target audio segment.

[0082] Step S420: If there is an inquiry statement in which the teacher asks the student a question in the semantic analysis result corresponding to the first target audio segment, determine that the raising hand action is a learning behavior.

[0083] It can be understood that the terminal device can determine the current student's action as a raising hand action through pose recognition, and then locate the action moment corresponding to the raising hand action, and extract the first target audio segment from the teaching audio and video. The first target audio segment is an audio segment with a corresponding duration before and after the raising hand action, and the corresponding duration can be set through presetting, such as set to 20s.

[0084] The terminal device performs semantic analysis on the first target audio segment to obtain the corresponding semantic analysis result. If it can be determined that there is an inquiry statement in which the teacher asks the student a question in the semantic analysis result, the terminal device can determine that the raising hand action is a learning behavior.

[0085] Exemplarily, at a certain moment, a student raises his hand. Correspondingly, the terminal device selects the audio segment of 20s before and after the raising hand action as the first target audio segment, and then performs semantic analysis. The terminal device can determine whether there is an inquiry statement in the first target audio segment according to the semantic analysis result. It can be imagined that, exemplarily, the inquiry statement can be "Does any classmate know what Ampere force is?" When performing semantic analysis, the terminal device determines that there is an inquiry statement in the audio segment through some keywords, such as "吗". Therefore, the terminal device can determine that the student's raising hand action is a learning behavior.

[0086] Thus, it can be seen that the solution of the present application can, on the basis of visually analyzing and determining the corresponding action of the student, combine the semantic analysis result of the corresponding audio segment to determine whether the action is a learning action, thereby more accurately identifying the learning behavior of the student, effectively reducing the occurrence of misjudgment, and improving the accuracy of teaching behavior judgment.

[0087] Figure 5 A flowchart of steps for determining teaching behavior provided in an embodiment of the present application is shown in the figure. In one embodiment, for teacher actions such as oral actions, the terminal device needs to analyze the audio clip to determine whether the action belongs to teaching behavior. The specific steps are as follows:

[0088] Step S510: When the teacher's action is a spoken action, a second target audio segment is selected based on the action moment corresponding to the spoken action.

[0089] Step S520: semantically analyze the teacher's speech content in the second target audio segment to determine whether there are any question statements from the teacher to the students or explanation statements involving knowledge point information.

[0090] Step S530: If a question sentence exists in the semantic analysis result corresponding to the second target audio segment, determine that the oral action is a questioning behavior.

[0091] Step S540: If there is an explanation sentence in the semantic analysis result corresponding to the second target audio segment, the oral action is determined to be a teaching behavior.

[0092] For the oral action of the teacher, the corresponding teaching behavior includes questioning behavior and lecturing behavior, that is, it is determined whether the oral action is a questioning behavior or a lecturing behavior. It can be imagined that the terminal device can also determine the oral action through visual analysis, such as after locating the position of the teacher, detecting the position of the teacher's mouth, and then detecting the change of the mouth shape to determine whether the current action is an oral action.

[0093] It is understandable that after determining that there is a spoken action, similarly, corresponding to the action moment of the spoken action, the terminal device extracts the corresponding audio segment from the teaching audio and video, that is, the second target audio segment. Therefore, the terminal device can perform semantic analysis on the second target audio segment. When there is a query sentence in the semantic analysis result corresponding to the second target audio segment, the terminal device can determine that the spoken action is a questioning behavior; when there is an explanation sentence in the semantic analysis result corresponding to the second target audio segment, the terminal device can determine that the spoken action is a teaching behavior.

[0094] It should be noted that for some teacher actions, such as writing actions, the terminal device can determine through visual analysis that the action is blackboard writing behavior in teaching behavior. For example, the terminal device can determine that the writing action performed by the teacher is blackboard writing behavior through the writing interface when the teacher performs writing actions in the video screen. For example, when the writing interface is a blackboard or teaching equipment, it can be determined that the writing action performed by the teacher is blackboard writing behavior.

[0095] Therefore, the present application scheme can determine whether the action belongs to teaching behavior based on the visual analysis of the teacher's corresponding action and the semantic analysis results of the corresponding audio clip, thereby more accurately identifying the teacher's teaching behavior, effectively reducing the occurrence of misjudgment, and improving the accuracy of teaching behavior judgment.

[0096] Figure 6 A schematic diagram of a teaching process timing diagram provided for an embodiment of the present application is shown in the figure, in which the time axis corresponds to the teaching audio and video, and there are multiple time nodes on the time axis to correspond to the knowledge point information and the teaching behavior. In order to better distinguish the knowledge point information from the teaching behavior, the knowledge point information is marked above the time axis in the figure, and the teaching behavior is marked below the time axis.

[0097] It is understandable that the terminal device generates a number of time nodes on the time axis according to the knowledge point information and the time of occurrence of the teaching behavior. In addition, the terminal device also sorts the time nodes along the time axis in accordance with the time sequence of the time nodes, thereby generating a teaching process sequence diagram. In the teaching process sequence diagram, each knowledge point information is associated with the teaching behavior at the corresponding time at the same time node.

[0098] Therefore, the terminal device can intuitively reflect the teacher's teaching time allocation for each knowledge point by generating a teaching process timing diagram, thereby helping teachers to allocate teaching time more reasonably, thereby assisting teachers in improving teaching quality.

[0099] Figure 7 A flowchart of the steps for determining the learning engagement of students provided in one embodiment of the present application. In some embodiments, the terminal device determines the learning engagement based on the sampling time, such as referring to the head-up rate statistics at the sampling time, and then determining the corresponding head-up rate curve to determine the learning engagement of students, so as to facilitate the analysis of teaching quality. Figure 7 As shown, the specific steps are as follows:

[0100] Step S710: based on the learning behaviors occurring at each sampling moment in the teaching video, the head-up rate at each sampling moment is counted to form a head-up rate curve that is continuous in time series, and the head-up rate curve is used to represent the learning engagement.

[0101] Step S720: Locate the head-up rate at each time node on the head-up rate curve to associate the learning engagement with the knowledge point information and the teaching behavior in time.

[0102] It is understandable that, for the case where the sampling moment is determined by a fixed sampling interval, the terminal device detects the student's learning behavior at the sampling moment, such as by detecting the student's head-up action in the video screen, and counting the head-up rate at each sampling moment to form a corresponding head-up rate curve to represent the student's learning engagement. Of course, for a head-up rate curve that is continuous in time series, the terminal device can generate a smooth curve as the head-up rate curve through the head-up rate at each sampling moment to represent its continuity in time series.

[0103] Moreover, the head-up rate curve is also associated with the time node, thereby temporally associating the knowledge point information and the teaching behavior. For example, a corresponding mark is generated at the position of the corresponding time node on the head-up rate curve to facilitate users to view the corresponding head-up rate. For example, when the user hovers the corresponding mouse cursor over the mark or clicks the mark, the corresponding head-up rate is displayed.

[0104] In addition, for the case where the sampling moment is the moment of the time node, the terminal device detects the student's learning behavior at the sampling moment, such as by detecting the student's head-up action in the video screen, and counting the head-up rate at each sampling moment to form a corresponding head-up rate curve to represent the student's learning engagement.

[0105] In one embodiment, for some target time nodes, the present application scheme also determines the learning engagement by combining the hand-raising rate, that is, the terminal device can combine the hand-raising rate and the head-up rate to make a more accurate judgment on the learning engagement of students in some scenarios, which is more helpful for the analysis of teaching quality.

[0106] It can be understood that for the statistics of hand-raising rate, similarly, the terminal device detects the hand-raising actions of students in the video screen, such as recognizing the students' postures, to determine the number of students who raise their hands, and then counts the hand-raising rate. Of course, the statistical moment corresponds to the target time node.

[0107] The target time node may be the time node corresponding to the teacher's questioning behavior. At this target time node, the terminal device can determine the student's learning engagement by combining the head-up rate and the hand-raising rate. For example, at this moment, the product of the head-up rate and the hand-raising rate and the corresponding weight is used as the learning engagement, while at other moments, the head-up rate is used as the learning engagement, thereby determining the corresponding learning engagement curve to facilitate teachers to view students' learning engagement levels.

[0108] Figure 8 A schematic diagram of a display window provided for an embodiment of the present application, as shown in the figure, in the display window, the upper part of the display window is used as the display area for displaying the video screen corresponding to the teaching audio and video and the curve corresponding to the learning engagement, while the lower part of the display window displays the teaching process timing diagram.

[0109] The figure shows an example of establishing associations between teaching audio and video, learning engagement, and teaching process timing diagram in the display window. As shown in the figure, when the user controls the mouse and clicks the cursor on the time node corresponding to the teacher's teaching of induced electromotive force, a pop-up window is displayed on the curve corresponding to the learning engagement to indicate the learning engagement at the time node, and the teaching audio and video will also jump to the screen corresponding to the moment associated with the time node. It is conceivable that in some embodiments, the above response can also be triggered when the cursor corresponding to the mouse hovers over the time node.

[0110] In addition, the user can play or pause the teaching audio and video by clicking in the display window. A mark is also provided on the curve corresponding to the learning engagement, and the mark corresponds to the time node, so that the user can view the learning engagement corresponding to the time node.

[0111] Fig. 9 This is a structural diagram of a teaching behavior analysis device provided in an embodiment of the present application, which is used to execute the teaching behavior analysis method provided in the above embodiment, and has functional modules and beneficial effects corresponding to the execution method. As shown in the figure, the device includes a knowledge point extraction module 801, a behavior analysis module 802, an image generation module 803, a data association module 804 and a window generation module 805.

[0112] The knowledge point extraction module 801 is configured to obtain the course information of the current classroom, and perform semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain the knowledge point information associated with the classroom teaching content;

[0113] The behavior analysis module 802 is configured to determine the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student based on the analysis results of the visual analysis and semantic analysis of the teaching audio and video;

[0114] The image generation module 803 is configured to determine the time nodes corresponding to each knowledge point information and each teaching behavior based on the time axis of the teaching audio and video, and generate a teaching process timing diagram;

[0115] The data association module 804 is configured to determine the learning engagement of students at each sampling moment according to the time axis of the teaching audio and video and the learning behavior, and associate the learning engagement with the corresponding knowledge point information and the time nodes of each teaching behavior;

[0116] The window generation module 805 is configured to generate and display a corresponding display window, which displays teaching audio and video, learning engagement and a teaching process timing diagram.

[0117] On the basis of the above embodiment, a teaching device for displaying courseware is configured in the classroom scene, and course information is recorded in the teaching device. The knowledge point extraction module 801 is further configured as follows:

[0118] Obtain course information from the teaching device, including the courseware currently being used, corresponding to the teaching subject, and the corresponding subject chapter;

[0119] Based on the audio data of the teaching audio and video, identify the speaker to determine the identity of the speaker and the corresponding audio clip of the speaker;

[0120] In the case where the speaker is a teacher, keywords are extracted from the audio clip in which the speaker is the teacher based on the knowledge point content corresponding to the subject chapter, and the knowledge point information is determined based on the keywords.

[0121] Based on the above embodiment, the knowledge point extraction module 801 is further configured as follows:

[0122] When there are multiple different keywords in an audio clip, the extracted keywords are filtered based on the temporal order of the knowledge point content, and the target keywords are locked.

[0123] Based on the above embodiment, the behavior analysis module 802 is further configured as follows:

[0124] Perform target detection on the collected video images to locate the teachers and students in the video images;

[0125] Based on a preset gesture recognition algorithm, determine the student action and the teacher action, and determine the first action moment of the student action and the second action moment of the teacher action;

[0126] Combined with the semantic analysis result of the first audio segment, it is determined that the student action is a learning behavior, and the first audio segment is an audio segment corresponding to the target duration of the first action moment;

[0127] Combined with the semantic analysis result of the second audio segment, it is determined that the teacher's action is a teaching behavior, and the second audio segment is an audio segment corresponding to the target duration containing the second action moment.

[0128] Based on the above embodiment, the student action includes a hand-raising action, and the behavior analysis module 802 is further configured as follows:

[0129] In the case where the student action is a hand-raising action, selecting a first target audio segment based on the action moment corresponding to the hand-raising action;

[0130] If there is a question sentence from the teacher to the student in the semantic analysis result corresponding to the first target audio segment, raising the hand is determined as a learning behavior.

[0131] On the basis of the above embodiment, the teacher action includes oral action, the teaching behavior includes questioning behavior and lecturing behavior, and the behavior analysis module 802 is further configured as follows:

[0132] In the case where the teacher's action is a spoken action, selecting a second target audio segment based on the action moment corresponding to the spoken action;

[0133] Semantic analysis is performed on the teacher's speech content in the second target audio clip to determine whether there are interrogation sentences from the teacher to the students or explanation sentences involving knowledge point information;

[0134] If there is a question sentence in the semantic analysis result corresponding to the second target audio segment, determining the oral action as a questioning behavior;

[0135] If there is an explanation sentence in the semantic analysis result corresponding to the second target audio segment, the oral action is determined to be a teaching behavior.

[0136] Based on the above embodiment, the image generation module 803 is further configured as follows:

[0137] According to the occurrence time of the knowledge point information and the teaching behavior, a number of time nodes are generated, and each knowledge point information is associated with the teaching behavior at the corresponding time at the same time node;

[0138] Combined with the temporal sequence of each time node, each time node is sorted along the time axis to form a teaching process timing diagram.

[0139] Based on the above embodiment, the data association module 804 is further configured as follows:

[0140] Based on the learning behaviors that appear at each sampling moment in the teaching video, the head-up rate at each sampling moment is counted to form a head-up rate curve that is continuous in time series, and the head-up rate curve is used to represent the learning engagement;

[0141] The head-up rate at each time node is located on the head-up rate curve to associate learning engagement with knowledge point information and teaching behavior in time.

[0142] Based on the above embodiment, the data association module 804 is further configured as follows:

[0143] At the target time point corresponding to the teaching behavior being the questioning behavior, the corresponding hand raising rate is counted based on the video screen;

[0144] Combine the hand-raising rate and head-raising rate to determine the learning engagement level at the corresponding target time node.

[0145] It is worth noting that in the embodiment of the above-mentioned teaching behavior analysis device, the various functional modules are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the various functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0146] Fig.10 The structural diagram of the terminal device provided for an embodiment of the present application is used to execute the teaching behavior analysis method provided in the above embodiment, and has the functional modules and beneficial effects corresponding to the execution method. As shown in the figure, it includes a processor 901, a memory 902, an input device 903 and an output device 904. The number of processors 901 can be one or more, and a processor 901 is taken as an example in the figure; the processor 901, the memory 902, the input device 903 and the output device 904 can be connected by a bus or other means, and the figure takes the connection through a bus as an example. The memory 902, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the teaching behavior analysis method in the embodiment of the present application. The processor 901 executes the corresponding various functional applications and data processing by running the software programs, instructions and modules stored in the memory 902, that is, realizing the above-mentioned teaching behavior analysis method.

[0147] The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data recorded or created during use, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 902 may further include a memory remotely arranged relative to the processor 901, and these remotely arranged memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] The input device 903 can be used to input corresponding digital or character information to the processor 901, and to generate key signal input related to the user settings and function control of the device; the output device 904 can be used to send or display key signal output related to the user settings and function control of the device.

[0149] An embodiment of the present application also provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to perform relevant operations in the teaching behavior analysis method provided in any embodiment of the present application.

[0150] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0151] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0152] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A teaching behavior analysis method, characterized in that: Applied to a terminal device, the terminal device is communicatively connected with an audio and video acquisition component in a classroom scene to receive teaching audio and video acquired by the audio and video acquisition component, the method comprising: Obtaining the course information of the current classroom, and performing semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain knowledge point information related to the classroom teaching content; Determining the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student according to the analysis results of the visual analysis and the semantic analysis of the teaching audio and video; Based on the timeline of the teaching audio and video, determine the time nodes corresponding to each knowledge point information and each teaching behavior, and generate a teaching process timing diagram; Determine the learning engagement of the students at each sampling moment according to the timeline of the teaching audio and video and the learning behavior, and associate the learning engagement with the corresponding knowledge point information and the time nodes of each teaching behavior; A corresponding display window is generated and displayed, wherein the display window displays the teaching audio and video, the learning engagement and the teaching process timing diagram.

2. The teaching behavior analysis method according to claim 1, characterized in that: The classroom scene is provided with a teaching device for displaying courseware, and the teaching device records the course information; The obtaining of the course information of the current classroom and performing semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain knowledge point information related to the classroom teaching content include: Acquire the course information from the teaching device, the course information including the teaching subject and the corresponding subject chapter of the courseware currently being used; Based on the audio data of the teaching audio and video, identifying the speaker to determine the identity of the speaker and the audio segment corresponding to the speaker; In the case where the speaker is a teacher, keywords are extracted from the audio segment in which the speaker is a teacher based on the knowledge point content corresponding to the subject chapter, and the knowledge point information is determined based on the keywords.

3. The teaching behavior analysis method according to claim 2, characterized in that: The obtaining of the course information of the current classroom, and performing semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain knowledge point information related to the classroom teaching content, further includes: When the audio clip contains multiple different keywords, the extracted keywords are filtered based on the temporal order of the knowledge point content, and the target keywords are locked.

4. The teaching behavior analysis method according to claim 1 or 2, characterized in that: The determining of the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student based on the analysis results of the visual analysis and the semantic analysis of the teaching audio and video includes: Performing target detection on the collected video images to locate the teacher and students in the video images; Based on a preset gesture recognition algorithm, determine the student action and the teacher action, and determine a first action moment of the student action and a second action moment of the teacher action; In combination with the semantic analysis result of the first audio segment, determining that the student action is the learning behavior, the first audio segment is an audio segment corresponding to the target duration of the first action moment; Combined with the semantic analysis result of the second audio segment, it is determined that the teacher action is the teaching behavior, and the second audio segment is an audio segment corresponding to the target duration of the second action moment.

5. The teaching behavior analysis method according to claim 4, characterized in that: The student action includes raising a hand; The step of determining the student action as the learning behavior based on the semantic analysis result of the first audio segment, wherein the first audio segment is an audio segment corresponding to the target duration of the first action moment, includes: In the case where the student action is a hand-raising action, selecting a first target audio segment based on an action moment corresponding to the hand-raising action; If there is a question sentence from the teacher to the student in the semantic analysis result corresponding to the first target audio segment, the hand raising is determined as a learning behavior.

6. The teaching behavior analysis method according to claim 4, characterized in that: The teacher action includes oral action, and the teaching behavior includes questioning behavior and lecturing behavior; The step of combining the semantic analysis result of the second audio segment to determine that the teacher action is the teaching behavior, wherein the second audio segment is an audio segment corresponding to the target duration of the second action moment, includes: In the case where the teacher action is a spoken action, selecting a second target audio segment based on the action moment corresponding to the spoken action; Performing semantic analysis on the teacher's speech content in the second target audio segment to determine whether there are any interrogation sentences from the teacher to the students or explanation sentences involving the knowledge point information; If the question sentence exists in the semantic analysis result corresponding to the second target audio segment, determining the oral action as the questioning behavior; If the explanation sentence exists in the semantic analysis result corresponding to the second target audio segment, the oral action is determined to be the teaching behavior.

7. The teaching behavior analysis method according to claim 1, characterized in that: The method of determining the time nodes corresponding to each knowledge point information and each teaching behavior based on the time axis of the teaching audio and video, and generating a teaching process timing diagram, includes: According to the knowledge point information and the occurrence time of the teaching behavior, a plurality of time nodes are generated, and each knowledge point information is associated with the teaching behavior at the corresponding time at the same time node; Combined with the temporal sequence of each time node, each time node is sorted along the time axis to form the teaching process timing diagram.

8. The teaching behavior analysis method according to claim 1, characterized in that: The determining of the learning engagement of the students at each sampling moment according to the time axis of the teaching audio and video and the learning behavior, and associating the learning engagement with the corresponding knowledge point information and the time nodes of each teaching behavior, includes: Based on the learning behaviors occurring at each sampling moment in the teaching video, counting the head-raising rate at each sampling moment to form a head-raising rate curve that is continuous in time series, and using the head-raising rate curve to represent the learning engagement; The head-up rate at each time node is located on the head-up rate curve to associate the learning engagement with the knowledge point information and the teaching behavior in time.

9. The teaching behavior analysis method according to claim 8, characterized in that: Also includes: At a target time point corresponding to when the teaching behavior is a questioning behavior, counting a corresponding hand-raising rate based on the video screen; The learning engagement corresponding to the target time node is determined by combining the hand-raising rate and the head-raising rate.

10. A teaching behavior analysis device, characterized in that: Applied to a terminal device, the terminal device is connected to the audio and video acquisition component in the classroom scene for communication to receive the teaching audio and video acquired by the audio and video acquisition component, the device comprises: A knowledge point extraction module is configured to obtain the course information of the current classroom, and perform semantic analysis on the audio data of the teaching audio and video in combination with the course information to obtain the knowledge point information related to the classroom teaching content; A behavior analysis module, configured to determine the teaching behavior corresponding to the teacher and the learning behavior corresponding to the student according to the analysis results of the visual analysis and semantic analysis of the teaching audio and video; An image generation module is configured to determine the time nodes corresponding to each knowledge point information and each teaching behavior based on the time axis of the teaching audio and video, and generate a teaching process timing diagram; A data association module is configured to determine the learning engagement of the students at each sampling moment according to the time axis of the teaching audio and video and the learning behavior, and associate the learning engagement with the corresponding knowledge point information and the time nodes of each teaching behavior; The window generation module is configured to generate and display a corresponding display window, wherein the display window displays the teaching audio and video, the learning engagement and the teaching process timing diagram.

11. A terminal device, characterized in that: include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the teaching behavior analysis method as described in any one of claims 1-9.

12. A storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, they are used to perform the teaching behavior analysis method according to any one of claims 1 to 9.

Images