Classroom content analysis method, system and equipment based on AI and storage medium

Abstract

The invention discloses an AI-based classroom content analysis method, system and device, and a storage medium. The method comprises the following steps: acquiring classroom video and voice data in real time; identifying and judging a current classroom scene by using the first multi-modal fusion and scene identification model; according to a preset classroom scene-interaction object mapping rule, determining an interaction object correspondingly mapped by the current classroom scene; utilizing a second multi-mode fusion and effective interaction identification model to extract behavior characteristics of the student, and judging whether the current student behavior is effective interaction; counting the ratio of the effective interaction duration to the total detection duration, and calculating the classroom concentration of the students; based on the interaction quality evaluation model, performing weighted calculation to obtain an interaction quality index of the student; and automatically generating an analysis report about the class concentration level and the interaction quality level according to a preset level evaluation standard. According to the method, accurate quantitative analysis of the concentration degree and the interaction quality of the students in the classroom can be realized, and the classroom analysis report is automatically generated.

Description

Technical Field

[0001] This application relates to the application of artificial intelligence in the field of education, and specifically to a method, system, device, and storage medium for analyzing classroom content based on AI. Background Art

[0002] Currently, with the development of artificial intelligence technology, the education industry has begun to attempt to use AI technology to improve teaching quality and efficiency, especially in the analysis of classroom content, and there have been preliminary explorations. These technologies can identify and transcribe speech, generate course summaries, and provide basic classroom interaction statistics, which play a certain role in improving teaching quality.

[0003] Existing classroom content analysis mainly relies on manual observation and simple automatic speech recognition (ASR) technology, supplemented by basic data statistical analysis, such as the recording of student concentration and interaction participation. Manual observation is that teachers or relevant personnel directly observe the performance and behavior of students in the classroom and record the participation of students and classroom reactions. Automatic speech recognition technology converts the speech in the classroom into text for subsequent analysis and processing. Basic data statistical analysis mainly conducts simple statistics and calculations on indicators such as student concentration and interaction participation to provide some basic data support for teaching evaluation.

[0004] However, these solutions have many limitations, including the inability to deeply and accurately understand classroom content, the lack of detailed analysis of classroom concentration and interaction quality, and the inability to automatically generate comprehensive intelligent analysis reports. And due to the limitations of simple automatic speech recognition technology itself, it is difficult to accurately identify and transcribe complex classroom speech, resulting in many errors in obtaining classroom content through simple speech transcription, which affects the accuracy of subsequent analysis; using simple data statistics, it is impossible to deeply explore the connotation of classroom content and the learning status of students, and it is difficult to scientifically and carefully evaluate classroom concentration and interaction quality; using manual data collation and analysis consumes a lot of time and energy, and the generated reports are not comprehensive and in-depth enough to meet the actual needs of education administrators and teachers. Summary of the Invention

[0005] In order to achieve precise quantitative analysis of students' classroom concentration and interaction quality and automatically generate classroom analysis reports; this application provides a method, system, device, and storage medium for analyzing classroom content based on AI.

[0006] In a first aspect, this application provides a method for analyzing classroom content based on AI, including: Real-time collection of classroom video and voice data; Using the first multi-modal fusion and scene recognition model constructed based on deep learning, analyze the current classroom video and voice data, extract video features and voice features, and identify and determine the current classroom scene; According to the preset classroom scene-interaction object mapping rule, determine the interaction object corresponding to the currently recognized classroom scene; Using the second multi-modal fusion and effective interaction recognition model constructed based on deep learning, analyze the current classroom video and voice data, extract the behavioral characteristics of students under the condition of the interaction object corresponding to the current classroom scene, and judge whether each current student behavior has an effective interaction with the interaction object corresponding to the current classroom scene; count the ratio of the effective interaction duration of each student behavior with the interaction object corresponding to the current classroom scene to the total detection duration, and calculate the classroom concentration of each student; Based on the interaction quality evaluation model, including according to the preset interaction quality evaluation criteria matched with the interaction object corresponding to the current classroom scene, at least including one of the interaction accuracy criterion, interaction initiative criterion, interaction depth criterion, and interaction emotional investment criterion, analyze and determine the classroom video and voice data in which each student behavior has an effective interaction with the interaction object corresponding to the current classroom scene, and calculate the interaction quality index of each student by weighted calculation; According to the preset level judgment criteria, automatically generate an analysis report on the classroom concentration level and interaction quality level based on the classroom concentration and interaction quality index of each student.

[0007] By adopting the above solution, multi-source data is collected, and the data is fused with the deep learning model to ensure the accuracy of subsequent analysis; the classroom scene is determined to map and obtain the interaction object, which helps to extract various behavioral characteristics of students to judge effective interaction and calculate concentration, realizing the precise quantitative analysis of classroom concentration; the interaction quality evaluation model is used to calculate the interaction quality index according to multiple criteria, improving the accuracy of interaction quality evaluation; according to the preset level judgment criteria, an analysis report on the classroom concentration level and interaction quality level is automatically generated.

[0008] Preferably, the automatically generating an analysis report on the classroom concentration level and interaction quality level according to the classroom concentration and interaction quality index of each student according to the preset level judgment criteria includes: Set the first influencing index of the preset judgment criteria for the classroom concentration of a single student; the first influencing index includes: classroom time, grade of the student in the classroom, class of the student in the classroom, distance position of the student in the classroom, physical condition of the student in the classroom, and classroom environment; Set the second influencing index of the preset judgment criteria for the interaction quality index of a single student; the second influencing index includes: classroom teaching subject, complexity of classroom teaching content, and classroom teaching interaction form; Analyze the collected classroom video and voice data according to the preset numerical calculation rules of the first impact index and the preset numerical calculation rules of the second impact index, and determine the corresponding impact index values of the first impact index and the corresponding impact index values of the second impact index; Match the impact coefficient of the preset evaluation criteria for the classroom concentration of a single student according to the corresponding impact index value of the first impact index, and adjust the preset evaluation criteria for the classroom concentration of a single student by multiplying the impact coefficient of the preset evaluation criteria for the classroom concentration of a single student; Match the impact coefficient of the preset evaluation criteria for the interactive quality index of a single student according to the corresponding impact index value of the second impact index, and adjust the preset evaluation criteria for the interactive quality index of a single student by multiplying the impact coefficient of the preset evaluation criteria for the interactive quality index of a single student; Determine the classroom concentration level of each student according to the adjusted preset evaluation criteria for the classroom concentration of a single student; Determine the interactive quality level of each student according to the adjusted preset evaluation criteria for the interactive quality index of a single student.

[0009] By adopting the above solution, consider and set the impact indicators related to classroom concentration and interactive quality index, determine the index values based on the collected classroom video and voice data, and then adjust the evaluation criteria by matching the impact coefficients, so as to more accurately determine the classroom concentration level and interactive quality level of each student according to the classroom concentration and interactive quality index of the students, and obtain a more accurate report.

[0010] Preferably, it further includes: Analyze the current classroom video and voice data according to the interaction object corresponding to the currently recognized classroom scene, and extract the preset abnormal behavior characteristics of each student's interaction object under the condition of the interaction object corresponding to the current classroom scene; Conduct a correlation analysis on the extracted behavior characteristics of the students and the preset abnormal behavior characteristics of the corresponding interaction objects, obtain the behavior characteristic pairs with a correlation coefficient greater than the first preset correlation coefficient and the behavior characteristic pairs with a correlation coefficient less than the second preset correlation coefficient, where the first preset correlation coefficient is greater than the second preset correlation coefficient; Reduce the weight of the student behavior characteristics in the behavior characteristic pairs with a correlation coefficient greater than the first preset correlation coefficient, and maintain the weight of the student behavior characteristics in the behavior characteristic pairs with a correlation coefficient less than the second preset correlation coefficient; Match the weight adjustment strategy of the preset interactive quality evaluation criteria according to the preset abnormal behavior characteristics of each student's interaction object extracted. Different preset abnormal behavior characteristics of interaction objects are preset with the corresponding weight adjustment strategies of the preset interactive quality evaluation criteria.

[0011] By adopting the above solution, considering that the classroom concentration and interaction quality index of students are affected by the interaction objects, and the abnormal behaviors of the interaction objects often lead to changes in students' behaviors, a correlation analysis is conducted between the students' behavior characteristics and the preset abnormal behavior characteristics, and the weights are adjusted to decouple the behavior characteristics and adjust the weights to reduce the interference with the students' behavior characteristics, so as to obtain more accurate classroom concentration and interaction quality indexes.

[0012] Preferably, it further includes: Determine whether the interaction objects corresponding to the currently recognized classroom scene are multiple objects. If it is determined that there are multiple objects, extract the preset abnormal behavior characteristics of each interaction object; Conduct a correlation analysis between the extracted behavior characteristics of the students and the preset abnormal behavior characteristics of each corresponding interaction object respectively to obtain different correlation coefficients; judge whether the difference between the maximum correlation coefficients among different correlation coefficients is greater than the preset coefficient difference. If it is greater, select to compare the maximum correlation coefficient with the first preset correlation coefficient and the second preset correlation coefficient; otherwise, select to compare the weighted average correlation coefficient with the first preset correlation coefficient and the second preset correlation coefficient.

[0013] By adopting the above solution, considering that there may be multiple interaction objects and each interaction object has an impact on it, a correlation analysis is conducted between the students' behavior characteristics and the preset abnormal behavior characteristics of each interaction object, and a suitable comparison method is adopted according to the different correlation coefficient differences, so as to more accurately evaluate the correlation between the students' behavior characteristics and the abnormal behavior characteristics.

[0014] Preferably, it further includes: Collect the after-class feedback data of each student, including: after-class exercise or assessment data; For each student, use the third multi-modal fusion and content recognition model constructed based on deep learning to analyze the classroom video and voice data, extract the video features and voice features under the condition of the interaction objects corresponding to the current classroom scene, and identify and judge the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction; Using the similarity comparison method, extract the relevant part of the after-class feedback data related to the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction from the after-class feedback data of the corresponding student; For each student, count the correct rates of exercises or assessments of the knowledge content involved in the class in the extracted partial after-class feedback data, and the correct rates of exercises or assessments of the knowledge content involved in classroom interactions; match the first correction index according to the correct rate of exercises or assessments of the knowledge content involved in the class, and use the first correction index to correct the classroom attention of the corresponding student; match the second correction index according to the correct rate of exercises or assessments of the knowledge content involved in classroom interactions, and use the second correction index to correct the interaction quality index of the corresponding student; among them, different correct rates of exercises or assessments of the knowledge content involved in the class are preset with corresponding first correction indexes; different correct rates of exercises or assessments of the knowledge content involved in classroom interactions are preset with corresponding second correction indexes.

[0015] By adopting the above solution, comprehensively consider the students' mastery of knowledge after class, and match the correction coefficients of classroom attention and interaction quality according to the mastery situation, so as to obtain more accurate classroom attention and interaction quality of students.

[0016] Preferably, according to the preset level evaluation criteria, the analysis report automatically generated based on the classroom attention and interaction quality indexes of each student further includes: In the order of the classroom video time, insert the classroom attention level and interaction quality level of each student at the corresponding moment into the corresponding student position in the video for display; Receive the instruction to obtain the situation of the specified student in the pre-specified period, and display the classroom video containing the classroom attention level and interaction quality level of the specified student obtained during the corresponding specified period in the analysis report.

[0017] By adopting the above solution, statistically analyze the classroom attention level and interaction quality level of the specified student during the specified period, and adopt the method of inserting the classroom attention level and interaction quality level into the video, which is convenient for teachers and educational administrators to obtain detailed information for specific periods and students, so as to more accurately conduct teaching quality evaluation and teaching decision-making.

[0018] Preferably, it further includes: Divide the total duration of the entire classroom video into each period, statistically analyze to obtain the time period with the highest density of the number of students whose classroom attention level is higher than the preset classroom attention level and the time period with the highest density of the number of students whose interaction quality level is higher than the preset interaction quality level, and display them in the analysis report.

[0019] By adopting the above solution, starting from the classroom attention and interaction quality of all students, determine the time period with the best classroom attention and interaction quality of all students, which helps teachers and educational administrators to understand the time periods when students have higher attention and interaction quality in the class.

[0020] In a second aspect, the present application provides an AI-based classroom content analysis system, comprising: A classroom data collection module for collecting classroom video and voice data in real time; A classroom scene recognition module for analyzing the current classroom video and voice data using a first multi-modal fusion and scene recognition model constructed based on deep learning, extracting video features and voice features, and identifying and determining the current classroom scene; An interaction object mapping module for determining the interaction object corresponding to the currently recognized classroom scene according to the preset classroom scene - interaction object mapping rules; A student concentration acquisition module for analyzing the current classroom video and voice data using a second multi-modal fusion and effective interaction recognition model constructed based on deep learning, extracting the behavioral characteristics of students under the condition of the interaction object corresponding to the current classroom scene, determining whether each current student behavior has an effective interaction with the interaction object corresponding to the current classroom scene; calculating the ratio of the duration of effective interaction between each student behavior and the interaction object corresponding to the current classroom scene to the total detection duration, and calculating the classroom concentration of each student; A student interaction quality acquisition module for, based on an interaction quality evaluation model, including analyzing and determining the classroom video and voice data of effective interaction between each student behavior and the interaction object corresponding to the current classroom scene according to a preset interaction quality evaluation criterion matching the interaction object corresponding to the current classroom scene, which at least includes one of an interaction accuracy criterion, an interaction initiative criterion, an interaction depth criterion, and an interaction emotional investment criterion, and calculating the interaction quality index of each student by weighted calculation; A classroom content analysis report generation module for automatically generating an analysis report on the classroom concentration level and interaction quality level according to the classroom concentration and interaction quality index of each student according to the preset level evaluation criterion.

[0021] By adopting the above solution, multi-source data is collected, the classroom scene is recognized and the interaction object is determined by combining with the deep learning model, multi-faceted behavioral characteristics of students are extracted to judge effective interaction, the classroom concentration and interaction quality index are calculated, and finally an analysis report on the classroom concentration level and interaction quality level is automatically generated, which can realize multi-dimensional analysis of the classroom and improve the accuracy and efficiency of classroom analysis.

[0022] In a third aspect, the present application provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the method as described above.

[0023] Fourthly, the present application provides a computer device, which includes a memory, a processor, and a program stored on the memory and executable thereon. When the program is executed by the processor, the steps of the above method are implemented.

[0024] In summary, the present application has the following beneficial effects: 1. By using the model constructed by deep learning to analyze classroom video and voice data, the depth of understanding of classroom content is improved; through the analysis of the deep learning model, the classroom scene and interaction objects are determined, the effective interaction of students is judged, the classroom concentration and interaction quality index are calculated, and an analysis report on the classroom concentration level and interaction quality level is automatically generated, improving the accuracy and efficiency of classroom analysis; 2. By combining multiple influencing indicators to adjust the evaluation criteria to determine the indicator values, and adjusting the evaluation criteria by matching the influence coefficient, the classroom concentration and interaction quality levels of students can be more accurately determined; 3. Adjust the weights of behavior characteristics and the weights of the preset interaction quality evaluation criteria according to the preset abnormal behavior characteristics of the interaction objects, and select a suitable correlation coefficient comparison method when multiple interaction objects are involved; at the same time, collect the students' after-class feedback data and compare it with the classroom content to correct the students' classroom concentration. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of the AI-based classroom content analysis method described in the specific embodiment; Figure 2 It is a schematic structural diagram of the AI-based classroom content analysis system described in the specific embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] As Figure 1 shown, the embodiment of the present application discloses an AI-based classroom content analysis method, and the specific steps include: S1. Real-time collect classroom video and voice data.

[0028] Specifically, to avoid large data errors in collecting classroom content solely through voice conversion, a variety of data collection devices such as microphone arrays, sensors, and camera devices are selected to collect classroom videos and capture voice data in real time in all directions. Among them, the microphone array can be composed of multiple voice signals and high-performance microphones. These microphones have the characteristics of high replaceable sensitivity and wide-frequency matrix microphone response, layout, etc.; sensors can be infrared sensors, pressure sensors, sound sensors, etc. installed in various corners of the classroom to detect information such as students' body temperature and movements; the camera device selects a camera with high resolution and wide-angle shooting capabilities to record classroom videos in all directions.

[0029] S2. Use the first multi-modal fusion and scene recognition model constructed based on deep learning to analyze the current classroom video and voice data, extract video features and voice features, and identify and judge the current classroom scene.

[0030] Specifically, considering different classroom scenes, the interaction objects of students are different, and the behavioral characteristics of students are different under different interaction object conditions. In order to accurately judge the concentration and interaction quality of students in the classroom based on their behavioral characteristics, it is necessary to pre-determine the current classroom scene.

[0031] Select to use deep learning to construct the first multi-modal fusion and scene recognition model. The input of this model is the classroom video and voice data collected in real time, and the output is the classroom scene. Specifically, input the currently collected classroom video and voice data into the model for feature extraction: including using CNN to analyze the spatio-temporal features of consecutive frames of pictures to extract video features, such as: the position of the teacher in the picture, the position of the student in the picture, the teacher's posture and expression, the student's posture and expression, the position of teaching equipment and the content displayed by teaching equipment, etc.; extract voice features, including using the beamforming technology of the microphone array to locate the sound source, combining voice activity detection to separate the teacher's and student's voices, inputting into the RNN classifier to identify scene keywords, obtaining acoustic features such as the teacher's voice acoustic features and the student's voice acoustic features, and text contents such as the teacher's output keywords and the student's output keywords; fuse the extracted video features and voice features for scene recognition.

[0032] Among them, the classroom video and voice data with historical annotations of the current classroom scene are used as training data to complete the model training of the first multi-modal fusion and scene recognition model. In this embodiment, the classroom scene specifically includes: teacher teaching scene, student free interaction scene, teacher-student interaction scene, and student self-study scene, etc. Correspondingly, the video features and voice features extracted under different scenes have different emphases. For example, the video features, voice features, and text features under the teacher teaching scene include: the teacher occupies the center of the screen, the students face the teacher, the teacher's voice accounts for 80%, and there are keywords such as "please see" and "attention"; the video features, voice features, and text features under the student free interaction scene include: students gather in groups, the students' voice accounts for 80%, and there are keywords such as "I think" and "how to do"; the video features, voice features, and text features under the teacher-student interaction scene include: the teacher is located in the student area, the teacher and the students look at each other, the students stand and mouth movements, the students' voice and the teacher's voice appear alternately; the video features, voice features, and text features under the student self-study scene include: the students lower their heads and face the textbooks, etc.

[0033] S3. According to the preset classroom scene-interaction object mapping rule, determine the interaction object corresponding to the currently recognized classroom scene in real time.

[0034] Specifically, preset the classroom scene-interaction object mapping rule, such as: under the teacher teaching scene, the interaction object is teacher-student; under the student free interaction scene, the interaction object is student-student; under the teacher-student interaction scene, the interaction object is teacher-student; under the student self-study scene, the interaction object is student-textbook / teaching equipment, etc.

[0035] According to the preset classroom scene-interaction object mapping rule, determine the interaction object corresponding to the currently recognized classroom scene in real time.

[0036] S4. Use the second multi-modal fusion and effective interaction recognition model constructed based on deep learning to analyze the current classroom video and voice data, judge whether each student's behavior under the condition of the interaction object corresponding to the current classroom scene produces an effective interaction with the interaction object corresponding to the current classroom scene, and calculate the student's classroom concentration.

[0037] For the purpose of concentration evaluation, a behavior matching based on interactive objects is designed. The core of the evaluation is whether the student's behavior can have an effective interaction with the standard interactive object in the current scenario. Under the corresponding mapped interactive object, a deep learning model is used to extract behavior features, specifically including: using the second multi-modal fusion and effective interaction recognition model constructed based on deep learning. Convolutional Neural Network (CNN) and Recurrent Neural Network can be used to build the model to improve the architecture that combines feature (RNN) extraction and scene recognition. For example, the second multi-modal fusion model uses a deep learning model. The input of this model is the classroom video and voice data and the interactive object corresponding to the current classroom scene, and the output is whether each student's behavior has an effective interaction with the interactive object corresponding to the current classroom scene. It is trained and generated by using the historical annotated classroom video and voice data on whether there is an effective interaction and the data of the interactive object corresponding to the classroom scene as training data.

[0038] Among them, using the second multi-modal fusion and effective interaction recognition model constructed based on deep learning to analyze the current classroom video and voice data, and extract the behavior features of students (involving interactions with the interactive object corresponding to the current classroom scene) under the condition of the interactive object corresponding to the current classroom scene, including: facial expression features, posture features, action features, and voice features, etc.; for different interactive objects corresponding to classroom scenes, the extracted behavior features of students have different focuses. For example, in the teacher's teaching scenario, the facial expression features of students focus on the proportion of the duration of looking at the teacher, the activity of expressions such as nodding / smiling / puzzled, etc., the posture features focus on the straightness of sitting posture facing the teacher, the angle of head raising and lowering, etc., the action features focus on turning pages / taking notes, etc., and the voice features include: the proportion of the voice duration of following the teacher's instructions; another example is that in the student free interaction scenario, the facial expression features of students focus on the proportion of the duration of looking at peers, the activity of expressions such as nodding / smiling / puzzled, etc., the posture features focus on the straightness of sitting posture facing the discussion object, etc., the action features focus on passing materials, etc., and the voice features include: the matching degree of the speaking duration and the speaking content with the discussion topic.

[0039] Therefore, using the second multi-modal fusion and effective interaction recognition model constructed based on deep learning to analyze the current classroom video and voice data, and judge whether each student's behavior under the condition of the interactive object corresponding to the current classroom scene has an effective interaction with the interactive object corresponding to the current classroom scene.

[0040] Statistically calculate the ratio of the duration of effective interaction between each student's behavior and the interactive object corresponding to the current classroom scene to the total detection duration, and calculate the classroom concentration of each student; the specific formula includes:

[0041] Among them, since the detection starts from the beginning of the class, the total detection duration refers to the duration of the current class progress.

[0042] S5. Calculate and obtain the interaction quality index of each student based on the interaction quality assessment model.

[0043] Specifically, preset the interaction quality assessment model; considering that there are differences in interactions in different classroom scenarios and with different interaction objects, different interaction quality assessment criteria are correspondingly set for different classroom scenarios and different interaction scenarios, including the preset interaction quality assessment criteria matching the interaction objects mapped corresponding to the current classroom scenario, including at least one of the interaction accuracy criterion, the interaction initiative criterion, the interaction depth criterion, and the interaction emotional investment criterion; among them, the evaluation index of the interaction accuracy criterion is content relevance, and the corresponding calculation methods include: the similarity between keywords in voice features and the current classroom theme, etc.; the evaluation index of the interaction initiative criterion is the number of interactions, and the corresponding calculation methods include: the number of times a student actively asks questions / speaks, etc.; the evaluation index of the interaction depth is the proportion of multi-round conversations, and the corresponding calculation methods include: the proportion of conversations with logical connections for more than N consecutive rounds, etc.; the evaluation index of the interaction emotional investment criterion is language emotional intensity, and the corresponding calculation methods include: calculating the proportion of positive emotions through an emotion classification model, etc.

[0044] In this embodiment, for the interaction objects mapped corresponding to the current classroom scenario being teacher-student or student-student, the preset interaction quality assessment criteria include: the interaction accuracy criterion, the interaction initiative criterion, the interaction depth criterion, and the interaction emotional investment criterion, etc.; for the interaction objects mapped corresponding to the current classroom scenario being student-textbook / teaching equipment, the preset interaction quality assessment criteria include: the interaction initiative criterion, the interaction emotional investment criterion, etc.

[0045] Thus, according to the preset interaction quality assessment criteria matching the interaction objects mapped corresponding to the current classroom scenario, analyze and determine the classroom video and voice data in which each student's behavior has an effective interaction with the interaction objects mapped corresponding to the current classroom scenario. The specific formula for calculating the interaction quality index is:

[0046] Among them, and The sum is 1, and for different preset interaction quality assessment criteria, and Correspondingly take 0.

[0047] S6. According to the preset grade judgment criteria, automatically generate an analysis report on the classroom concentration level and interaction quality level based on the classroom concentration and interaction quality index of each student.

[0048] Specifically, the preset level evaluation criteria include the evaluation criteria for classroom concentration and the evaluation criteria for interaction quality. The evaluation criteria for classroom concentration are used to grade the classroom concentration of each student, and the evaluation criteria for interaction quality levels are used to grade the interaction quality of each student, automatically generating an analysis report on classroom concentration levels and interaction quality levels.

[0049] In a specific embodiment, considering the factors that affect the classroom concentration and interaction quality index of each student in the classroom, in order to more accurately determine the classroom concentration level and interaction quality level of different students, the preset level evaluation criteria are adjusted. The method of automatically generating an analysis report on classroom concentration levels and interaction quality levels according to the preset level evaluation criteria based on the classroom concentration and interaction quality index of each student includes: Setting the first influencing indicators for the preset evaluation criteria of a single student's classroom concentration; the first influencing indicators include: classroom time, the grade of the student in the classroom, the class of the student in the classroom, the distance position of the student in the classroom, the physical condition of the student in the classroom, and the classroom environment, etc.; corresponding preset numerical calculation rules are set for each single first influencing indicator. For example, students' classroom concentration is higher during the period from 9 to 10 am during the day, and the corresponding preset value is higher. Accordingly, corresponding preset numerical matching is performed for each period within 24 hours of the classroom time. Correspondingly, the smaller the grade of the student in the classroom or the student is in Class B, the lower the classroom concentration of the corresponding student, and the lower the corresponding preset value. Accordingly, corresponding preset numerical matching is performed according to the grade of the student in the classroom and whether the student is in Class A or Class B in the classroom. Correspondingly, the farther the student in the classroom is from the blackboard, the lower the classroom concentration of the corresponding student, and the lower the corresponding preset value. Accordingly, corresponding preset numerical matching is performed according to the distance position of the student in the classroom. Correspondingly, the worse the physical condition of the student in the classroom (determined by judging body temperature through an infrared camera, judging whether the expression is painful through a camera device, etc.), the lower the classroom concentration of the corresponding student, and the lower the corresponding preset value. Accordingly, corresponding preset numerical matching is performed according to the physical condition of the student in the classroom. Correspondingly, the more classroom environmental noise (determined by extracting classroom noise characteristics), the lower the classroom concentration of the corresponding student, and the lower the corresponding preset value. Accordingly, corresponding preset numerical matching is performed according to the classroom environmental noise.

[0050] Set the second influencing indicators for the preset evaluation criteria of the individual student interaction quality index; the second influencing indicators include: classroom teaching subjects, complexity of classroom teaching content, forms of classroom teaching interaction, etc.; set corresponding preset numerical calculation rules for each individual second influencing indicator. For example, for the main teaching subject, the higher the student interaction quality, the higher the corresponding interaction quality index. Accordingly, make corresponding preset numerical matching settings for whether the classroom teaching subject is the main subject; for the complexity of classroom teaching content (which can be determined according to the accuracy of the course teaching content in the class schedule), the more complex the content, the lower the student interaction quality, and the lower the corresponding interaction quality index. Accordingly, make corresponding preset numerical matching settings for the complexity of classroom teaching content; for the form of remote classroom teaching interaction, the student interaction quality is even lower, and the corresponding interaction quality index is lower. Accordingly, make corresponding preset numerical matching settings for whether the classroom teaching interaction form is remote teaching.

[0051] According to the preset numerical calculation rules of the first influencing indicator and the preset numerical calculation rules of the second influencing indicator, analyze the collected classroom video and voice data to determine the corresponding influencing indicator values of the first influencing indicator and the corresponding influencing indicator values of the second influencing indicator; among them, the corresponding influencing indicator values of the first influencing indicator are all obtained by weighted calculation of the corresponding influencing indicator values of each individual first influencing indicator, and the corresponding influencing indicator values of the second influencing indicator are all obtained by weighted calculation of the corresponding influencing indicator values of each individual second influencing indicator.

[0052] Match the influencing coefficient of the preset evaluation criteria for individual student classroom concentration according to the corresponding influencing indicator values of the first influencing indicator, and adjust the preset evaluation criteria for individual student classroom concentration by multiplying the influencing coefficient of the preset evaluation criteria for individual student classroom concentration; among them, the corresponding influencing indicator values of different first influencing indicators are all set with the influencing coefficients of the preset evaluation criteria for classroom concentration that match them; match the influencing coefficient of the preset evaluation criteria for the individual student interaction quality index according to the corresponding influencing indicator values of the second influencing indicator, and adjust the preset evaluation criteria for the individual student interaction quality index by multiplying the preset evaluation criteria for the individual student interaction quality index. The corresponding influencing indicator values of different second influencing indicators are all set with the influencing coefficients of the preset evaluation criteria for the interaction quality index that match them.

[0053] According to the adjusted preset evaluation criteria for individual student classroom concentration, complete the determination of the classroom concentration level of each student; according to the adjusted preset evaluation criteria for the individual student interaction quality index, complete the determination of the interaction quality level of each student.

[0054] In a specific embodiment, considering that the classroom concentration and interaction quality of students are often highly correlated with the interaction object, if the interaction object exhibits abnormal behaviors, it will correspondingly affect the classroom concentration and interaction quality of students. To further improve the accuracy of the analysis of students' classroom concentration and interaction quality, the method further includes: Analyze the current classroom video and voice data based on the interaction object corresponding to the currently recognized classroom scene in real time, and extract the preset abnormal behavior characteristics of each student interaction object under the condition of the interaction object corresponding to the current classroom scene. For example, when the interaction object is a teacher, the preset abnormal behavior characteristics include: abnormal speech rate, voice interruption, anxious or tired facial expressions, abnormal movement, blackboard writing errors, and other teacher abnormal behavior characteristics indicating teacher fatigue; when the interaction object is a student, the preset abnormal behavior characteristics include: abnormal movement of students, abnormal voice intensity of students, and other student abnormal behavior characteristics indicating restlessness of the student group; when the interaction object is teaching materials / teaching equipment, the preset abnormal behavior characteristics include: abnormal projection screen, occlusion of blackboard writing content, interruption of equipment screen, and other abnormal behavior characteristics indicating occlusion of teaching materials content / failure of teaching equipment. Conduct a correlation analysis on the extracted behavior characteristics of students and the preset abnormal behavior characteristics of the corresponding interaction object. In this embodiment, the correlation analysis is calculated using the Pearson correlation coefficient. Obtain the behavior feature pairs with a correlation coefficient greater than the first preset correlation coefficient and the behavior feature pairs with a correlation coefficient less than the second preset correlation coefficient. The first preset correlation coefficient is greater than the second preset correlation coefficient. Reduce the weight of the student behavior characteristics in the behavior feature pairs with a correlation coefficient greater than the first preset correlation coefficient. For example, if the correlation between the teacher's abnormal speech rate and the student's frowning facial expression is greater than the first preset correlation coefficient, it indicates that the current student's facial representation is an interference item caused by the teacher's abnormal behavior, and reduce the weight of the facial expression feature in the concentration judgment. Another example: if the correlation between the occlusion of the blackboard writing content and the student's eye gaze deviation is greater than the first preset correlation coefficient, it indicates that the current student's lack of concentration is due to the occlusion of the blackboard writing, and reduce the weight of the eye gaze deviation facial expression feature in the concentration judgment. Keep the weight of the student behavior characteristics in the behavior feature pairs with a correlation coefficient less than the second preset correlation coefficient. The behavior feature pairs with a correlation coefficient less than the second preset correlation coefficient indicate that the concentration of students does not shift due to the abnormality of the interaction object, so the weight remains unchanged.

[0055] According to the preset abnormal behavior characteristics of each student interaction object extracted, match the weight adjustment strategy of the preset interaction quality evaluation criteria. For different interaction objects, there are preset weight adjustment strategies of the preset interaction quality evaluation criteria that match them. Specifically, it includes: the teacher abnormal behavior characteristics representing teacher fatigue correspond to reducing the weights of the interaction initiative standard and the interaction depth standard; the student abnormal behavior characteristics representing restlessness of the student group correspond to reducing the weight of the interaction depth standard; the abnormal behavior characteristics representing textbook content occlusion / teaching equipment failure correspond to reducing the weights of the interaction accuracy standard and the interaction emotional investment.

[0056] In addition, considering that there may be multiple user interaction objects, such as multiple student interactions, in order to avoid the simultaneous impact of multiple interaction objects on students' classroom concentration, the method further includes: determining whether the interaction objects corresponding to the currently recognized classroom scenario mapping are multiple objects; if it is determined that there are multiple interaction objects, extracting the preset abnormal behavior characteristics of each interaction object; respectively performing correlation analysis on the extracted student behavior characteristics and the preset abnormal behavior characteristics of each corresponding interaction object to obtain different correlation coefficients; judging whether the maximum correlation coefficient gap between different correlation coefficients is greater than the preset coefficient difference; if it is greater, then choose to compare the maximum correlation coefficient with the first preset correlation coefficient and the second preset correlation coefficient; otherwise, choose to compare the weighted average correlation coefficient with the first preset correlation coefficient and the second preset correlation coefficient.

[0057] In a specific embodiment, considering the problems existing in the judgment of classroom concentration and interaction quality, and correcting it in combination with the after-class feedback data, the method further includes: Collecting the after-class feedback data of each student, including: after-class exercise or assessment data.

[0058] For each student, using the third multi-modal fusion and content recognition model constructed based on deep learning, analyze the current classroom video and voice data, extract the video features and voice features under the conditions of the interaction objects corresponding to the current classroom scenario mapping, and identify and judge the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction; the third multi-modal fusion and content recognition model uses a deep learning model, the input of this model is the classroom video and voice data and the interaction objects corresponding to the current classroom scenario mapping, and the output is the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction, and it is trained and generated through the historical interaction objects corresponding to each student in the classroom scenario mapping, the classroom video and voice data, and the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction in the classroom video and voice data as training data.

[0059] Using the similarity comparison method, extract the relevant part of the after-class feedback data related to the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction from the after-class feedback data of the corresponding student; For each student, count the correct rates of exercises or assessments on the knowledge content covered in class and the correct rates of exercises or assessments on the knowledge content involved in classroom interactions in the extracted partial after-class feedback data; match the first correction index according to the correct rate of exercises or assessments on the knowledge content covered in class, and use the first correction index to correct the classroom concentration of the corresponding student; match the second correction index according to the correct rate of exercises or assessments on the knowledge content involved in classroom interactions, and use the second correction index to correct the interaction quality index of the corresponding student; among them, different correct rates of exercises or assessments on the knowledge content covered in class are preset with corresponding first correction indexes; different correct rates of exercises or assessments on the knowledge content involved in classroom interactions are preset with corresponding second correction indexes.

[0060] In a specific embodiment, in order to better assist teachers in understanding the classroom concentration and interaction quality of each student, the method of automatically generating an analysis report on the classroom concentration level and interaction quality level according to the classroom concentration and interaction quality index of each student according to the preset level evaluation criteria further includes: Insert the classroom concentration level and interaction quality level of each student at the corresponding moment into the corresponding student position in the video for display according to the chronological order of the classroom video. In order to facilitate teachers to view the classroom concentration and interaction quality for a specific period (such as: the period when important knowledge points are taught in the teaching plan) and specific students (such as: students with fluctuating grades in the recent period), receive a situation acquisition instruction for a specified student in a pre-specified period, and display the classroom video containing the classroom concentration level and interaction quality level of the specified student obtained in the corresponding specified period in the analysis report.

[0061] In addition, starting from the analysis requirements of the entire classroom teaching quality, the method further includes: Divide the full duration of the entire classroom video into each period, and statistically analyze to obtain the period with the highest density of the number of students whose classroom concentration level is higher than the preset classroom concentration level and the period with the highest density of the number of students whose interaction quality level is higher than the preset interaction quality level, and display them in the analysis report.

[0062] As Figure 2 shown, the embodiment of the present application discloses an AI-based classroom content analysis system, which specifically includes: A classroom data acquisition module 101 for real-time acquisition of classroom video and voice data; A classroom scene recognition module 102 for using a first multi-modal fusion and scene recognition model constructed based on deep learning to analyze the current classroom video and voice data, extract video features and voice features, and identify and judge the current classroom scene; The interaction object mapping module 103 is configured to determine the interaction object mapped to the currently recognized classroom scenario according to the preset classroom scenario - interaction object mapping rule; The student concentration acquisition module 104 is configured to analyze the current classroom video and voice data by using the second multi-modal fusion and effective interaction recognition model constructed based on deep learning, extract the behavior characteristics of students under the condition of the interaction object mapped to the current classroom scenario, and determine whether each current student behavior has an effective interaction with the interaction object mapped to the current classroom scenario; count the ratio of the duration of effective interaction between each student behavior and the interaction object mapped to the current classroom scenario to the total detection duration, and calculate the classroom concentration of each student; The student interaction quality acquisition module 105 is configured to, based on the interaction quality assessment model, including at least one of the preset interaction quality assessment criteria matched with the interaction object mapped to the current classroom scenario, such as the interaction accuracy criterion, the interaction initiative criterion, the interaction depth criterion, and the interaction emotional investment criterion, analyze and determine the classroom video and voice data in which each student behavior has an effective interaction with the interaction object mapped to the current classroom scenario, and calculate the interaction quality index of each student by weighted calculation; The classroom content analysis report generation module 106 is configured to automatically generate an analysis report on the classroom concentration level and the interaction quality level according to the classroom concentration and the interaction quality index of each student according to the preset level judgment criteria.

[0063] In a specific embodiment, the system further includes: a student concentration acquisition optimization module 107, configured to analyze the current classroom video and voice data according to the interaction object mapped to the currently recognized classroom scenario, and extract the preset abnormal behavior characteristics of each student interaction object under the condition of the interaction object mapped to the current classroom scenario; perform a correlation analysis on the extracted behavior characteristics of the students and the preset abnormal behavior characteristics of the corresponding interaction objects to obtain the behavior characteristic pairs with a correlation coefficient greater than the first preset correlation coefficient and the behavior characteristic pairs with a correlation coefficient less than the second preset correlation coefficient, where the first preset correlation coefficient is greater than the second preset correlation coefficient; reduce the weight of the student behavior characteristics in the behavior characteristic pairs with a correlation coefficient greater than the first preset correlation coefficient, and maintain the weight of the student behavior characteristics in the behavior characteristic pairs with a correlation coefficient less than the second preset correlation coefficient; It is also used to determine whether the interaction objects corresponding to the currently recognized classroom scene are multiple objects; if it is determined that there are multiple objects, the preset abnormal behavior characteristics of each interaction object are extracted; the behavior characteristics of the extracted students are respectively analyzed for correlation with the preset abnormal behavior characteristics of each corresponding interaction object to obtain different correlation coefficients; it is determined whether the difference between the maximum correlation coefficients among the different correlation coefficients is greater than a preset coefficient difference. If it is greater, the maximum correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient; otherwise, the weighted average correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient.

[0064] The student interaction quality acquisition and optimization module 108 is used to match the weight adjustment strategy of the preset interaction quality evaluation standard according to the preset abnormal behavior characteristics of each student interaction object. The preset abnormal behavior characteristics of different interaction objects are preset with weight adjustment strategies that match them for the preset interaction quality evaluation standard.

[0065] In a specific embodiment, the system further includes: a student concentration and interaction quality correction module 109, which is used to collect the after-class feedback data of each student, including: after-class exercise or assessment data; for each student, using a third multi-modal fusion and content recognition model constructed based on deep learning, analyzing the classroom video and voice data, extracting the video features and voice features under the condition of the interaction object corresponding to the current classroom scene, and identifying and judging the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction; using a similarity comparison method to extract part of the after-class feedback data related to the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction from the corresponding student's after-class feedback data; for each student, counting the exercise or assessment correct rate of the knowledge content involved in the classroom and the exercise or assessment correct rate of the knowledge content involved in the classroom interaction in the extracted part of the after-class feedback data; matching the first correction index according to the exercise or assessment correct rate of the knowledge content involved in the classroom, and using the first correction index to correct the classroom concentration of the corresponding student; matching the second correction index according to the exercise or assessment correct rate of the knowledge content involved in the classroom interaction, and using the second correction index to correct the interaction quality index of the corresponding student; wherein, different exercise or assessment correct rates of the knowledge content involved in the classroom are preset with first correction indexes that match them; different exercise or assessment correct rates of the knowledge content involved in the classroom interaction are preset with second correction indexes that match them.

[0066] The embodiment of the present application also discloses a computer-readable storage medium.

[0067] Specifically, the computer-readable storage medium stores a computer program that can be loaded and executed by a processor, such as the above-described AI-based classroom content analysis method. The computer-readable storage medium includes, for example, various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0068] An embodiment of the present application also discloses a computer device.

[0069] Specifically, the computer device includes a memory and a processor. The memory stores a computer program that can be loaded and executed by the processor, such as the above-described AI-based classroom content analysis method.

[0070] The above are all preferred embodiments of the present application. The protection scope of the present application is not limited by this. Any feature disclosed in this specification (including the abstract and drawings), unless specifically described, can be replaced by other equivalent or similar-purpose alternative features. That is, unless specifically described, each feature is only an example of a series of equivalent or similar features.

Claims

1. An AI-based classroom content analysis method, characterized in that, Including: Real-time collecting classroom video and voice data; Using a first multi-modal fusion and scene recognition model constructed based on deep learning to analyze the current classroom video and voice data, extract video features and voice features, and identify and determine the current classroom scene; According to the preset classroom scene-interaction object mapping rules, determining the interaction object corresponding to the currently recognized classroom scene; Using a second multi-modal fusion and effective interaction recognition model constructed based on deep learning to analyze the current classroom video and voice data, extract the behavioral characteristics of students under the conditions of the interaction object corresponding to the current classroom scene, and judge whether each current student behavior has an effective interaction with the interaction object corresponding to the current classroom scene; Statistical ratio of the effective interaction duration of each student behavior with the interaction object corresponding to the current classroom scene to the total detection duration, and calculating the classroom concentration of each student; Based on the interaction quality assessment model, including analyzing and determining the classroom video and voice data in which each student behavior has an effective interaction with the interaction object corresponding to the current classroom scene according to the preset interaction quality assessment criteria matched with the interaction object corresponding to the current classroom scene, at least including one of the interaction accuracy criteria, interaction initiative criteria, interaction depth criteria, and interaction emotional investment criteria, and calculating the interaction quality index of each student by weighted calculation; According to the preset grade evaluation criteria, automatically generating an analysis report on the classroom concentration level and interaction quality level based on the classroom concentration and interaction quality index of each student.

2. The AI-based classroom content analysis method according to claim 1, wherein The automatically generating an analysis report on the classroom concentration level and interaction quality level according to the preset grade evaluation criteria based on the classroom concentration and interaction quality index of each student includes: Setting a first influencing index for the preset evaluation criteria of the classroom concentration of a single student; the first influencing index includes: classroom time, grade of the student in the classroom, class of the student in the classroom, distance position of the student in the classroom, physical condition of the student in the classroom, and classroom environment; Setting a second influencing index for the preset evaluation criteria of the interaction quality index of a single student; the second influencing index includes: classroom teaching subject, complexity of classroom teaching content, and classroom teaching interaction form; According to the preset numerical calculation rules of the first influencing index and the preset numerical calculation rules of the second influencing index, analyzing the collected classroom video and voice data, and determining the corresponding influencing index values of the first influencing index and the influencing index values corresponding to the second influencing index; Matching the corresponding influencing index values of the first influencing index with the influencing coefficient of the preset evaluation criteria of the classroom concentration of a single student, and adjusting the preset evaluation criteria of the classroom concentration of a single student by multiplying the influencing coefficient of the preset evaluation criteria of the classroom concentration of a single student; matching the corresponding influencing index values of the second influencing index with the influencing coefficient of the preset evaluation criteria of the interaction quality index of a single student, and adjusting the preset evaluation criteria of the interaction quality index of a single student by multiplying the preset evaluation criteria of the interaction quality index of a single student; Determine the classroom concentration level of each student according to the adjusted preset judgment criteria for the classroom concentration of a single student; determine the interactive quality level of each student according to the adjusted preset judgment criteria for the interactive quality index of a single student.

3. The AI-based classroom content analysis method according to claim 1, wherein It also includes: Analyze the current classroom video and voice data according to the interaction objects corresponding to the currently recognized classroom scenarios in real time, and extract the preset abnormal behavior characteristics of each student's interaction objects under the conditions of the interaction objects corresponding to the currently recognized classroom scenarios. Conduct a correlation analysis on the extracted behavioral characteristics of the students and the preset abnormal behavioral characteristics of the corresponding interaction objects to obtain pairs of behavioral characteristics with a correlation coefficient greater than the first preset correlation coefficient and pairs of behavioral characteristics with a correlation coefficient less than the second preset correlation coefficient, where the first preset correlation coefficient is greater than the second preset correlation coefficient; reduce the weight of the behavioral characteristics of the students in the pairs of behavioral characteristics with a correlation coefficient greater than the first preset correlation coefficient, and maintain the weight of the behavioral characteristics of the students in the pairs of behavioral characteristics with a correlation coefficient less than the second preset correlation coefficient. Match the preset abnormal behavior characteristics of each student's interaction objects to the weight adjustment strategy of the preset interactive quality evaluation criteria. Different preset abnormal behavior characteristics of interaction objects are preset with weight adjustment strategies of the preset interactive quality evaluation criteria that match them.

4. The AI-based classroom content analysis method according to claim 3, wherein It also includes: Determine whether the interaction objects corresponding to the currently recognized classroom scenarios in real time are multiple objects. If it is determined that there are multiple objects, extract the preset abnormal behavior characteristics of each interaction object. Conduct a correlation analysis on the extracted behavioral characteristics of the students and the preset abnormal behavioral characteristics of each corresponding interaction object respectively to obtain different correlation coefficients; determine whether the maximum correlation coefficient gap between different correlation coefficients is greater than the preset coefficient difference; if it is greater, then choose to compare the maximum correlation coefficient with the first preset correlation coefficient and the second preset correlation coefficient. Otherwise, choose to compare the weighted average correlation coefficient with the first preset correlation coefficient and the second preset correlation coefficient.

5. The AI-based classroom content analysis method according to claim 1, wherein It also includes: Collect the after-class feedback data of each student, including: after-class exercise or assessment data. For each student, use the third multi-modal fusion and content recognition model constructed based on deep learning to analyze the classroom video and voice data, extract the video features and voice features under the conditions of the interaction objects corresponding to the currently recognized classroom scenarios, and identify and judge the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction. Use the similarity comparison method to extract part of the after-class feedback data related to the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction from the after-class feedback data of the corresponding student. For each student, count the correct rates of exercises or assessments of the knowledge content involved in the class in the extracted partial after-class feedback data, and the correct rates of exercises or assessments of the knowledge content involved in classroom interactions; match the first correction index according to the correct rate of exercises or assessments of the knowledge content involved in the class, and use the first correction index to correct the classroom concentration of the corresponding student; match the second correction index according to the correct rate of exercises or assessments of the knowledge content involved in classroom interactions, and use the second correction index to correct the interaction quality index of the corresponding student; among them, different correct rates of exercises or assessments of the knowledge content involved in the class are preset with corresponding first correction indexes; different correct rates of exercises or assessments of the knowledge content involved in classroom interactions are preset with corresponding second correction indexes.

6. The AI-based classroom content analysis method according to claim 2, wherein The automatically generating an analysis report on the classroom concentration level and interaction quality level according to the classroom concentration and interaction quality index of each student according to the preset level evaluation criteria further includes: Insert the classroom concentration level and interaction quality level of each student at the corresponding moment into the corresponding student position in the video for display according to the chronological order of the classroom video; Receive the instruction to obtain the situation of a specified student in a pre-specified period, and display the classroom video containing the classroom concentration level and interaction quality level situation of the specified student obtained in the corresponding specified period in the analysis report.

7. The AI-based classroom content analysis method according to claim 2, wherein It also includes: Divide the entire duration of the classroom video into each period, statistically analyze to obtain the time period with the highest density of the number of students whose classroom concentration level is higher than the preset classroom concentration level and the time period with the highest density of the number of students whose interaction quality level is higher than the preset interaction quality level among all students, and display them in the analysis report.

8. An AI-based classroom content analysis system, characterized in that, It includes: A classroom data collection module for collecting classroom video and voice data in real time; A classroom scene recognition module for analyzing the current classroom video and voice data, extracting video features and voice features, and identifying and judging the current classroom scene by using the first multi-modal fusion and scene recognition model constructed based on deep learning; An interaction object mapping module for determining the interaction object mapped corresponding to the currently recognized classroom scene according to the preset classroom scene-interaction object mapping rule; A student concentration acquisition module for analyzing the current classroom video and voice data by using the second multi-modal fusion and effective interaction recognition model constructed based on deep learning, extracting the behavioral characteristics of students under the condition of the interaction object mapped corresponding to the current classroom scene, and judging whether the current behavior of each student has an effective interaction with the interaction object mapped corresponding to the current classroom scene; count the ratio of the duration of effective interaction between the behavior of each student and the interaction object mapped corresponding to the current classroom scene to the total detection duration, and calculate the classroom concentration of each student; A student interaction quality acquisition module, which is used to analyze and determine the classroom video and voice data in which each student behavior has an effective interaction with the interaction object corresponding to the current classroom scenario based on an interaction quality assessment model, including a preset interaction quality assessment standard that matches the interaction object corresponding to the current classroom scenario, and at least includes one of the interaction accuracy standard, the interaction initiative standard, the interaction depth standard, and the interaction emotional investment standard, and weighted calculation is used to obtain the interaction quality index of each student; A classroom content analysis report generation module, which is used to automatically generate an analysis report on the classroom concentration level and the interaction quality level according to the classroom concentration and the interaction quality index of each student according to a preset level judgment standard.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the method described in any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a program stored on the memory and executable thereon. When the program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 7.

Images