AI-based classroom content analysis method, system, equipment and storage medium

Abstract

The present application discloses an AI-based classroom content analysis method, system, device and storage medium, the method comprising: real-time collection of classroom video and voice data; using a first multimodal fusion and scene recognition model to identify and determine the current classroom scene; determining the interactive object mapped to the current classroom scene according to a preset classroom scene-interaction object mapping rule; using a second multimodal fusion and effective interaction recognition model to extract student behavior characteristics and determine whether the current student behavior is an effective interaction; calculating the student's classroom concentration by calculating the ratio of effective interaction time to total detection time; obtaining the student's interaction quality index through weighted calculation based on an interaction quality assessment model; and automatically generating an analysis report on the classroom concentration level and interaction quality level according to a preset grade evaluation standard. The present application can achieve accurate quantitative analysis of students' classroom concentration and interaction quality, and automatically generate a classroom analysis report.

Description

Technical Field

[0001] This application relates to the application of artificial intelligence in the field of education, specifically to an AI-based classroom content analysis method, system, device and storage medium. Background Art

[0002] With the development of artificial intelligence (AI), the education industry is beginning to explore leveraging AI to improve teaching quality and efficiency, particularly in classroom content analysis. These technologies can recognize and transcribe speech, generate course summaries, and provide basic classroom interaction statistics, contributing to improved teaching quality.

[0003] Existing classroom content analysis primarily relies on manual observation and simple automatic speech recognition (ASR) technology, supplemented by basic statistical analysis, such as recording student focus and interactive engagement. Manual observation involves teachers or other personnel directly observing student performance and behavior in class, recording student participation and responses. Automatic speech recognition technology converts classroom speech into text for subsequent analysis and processing. Basic statistical analysis primarily involves simple statistics and calculations on indicators such as student focus and interactive engagement, providing basic data support for teaching evaluation.

[0004] However, these solutions have numerous limitations, including an inability to provide a deep and accurate understanding of classroom content, a lack of detailed analysis of classroom focus and interaction quality, and an inability to automatically generate comprehensive intelligent analysis reports. Furthermore, due to the inherent limitations of simple automatic speech recognition technology, it is difficult to accurately identify and transcribe complex classroom speech, resulting in numerous errors in classroom content captured through simple speech transcription, which affects the accuracy of subsequent analysis. Using simple data statistics fails to deeply explore the connotations of classroom content and students' learning status, making it difficult to conduct scientific and detailed assessments of classroom focus and interaction quality. Manual data compilation and analysis consumes significant time and effort, and the resulting reports are insufficiently comprehensive and in-depth, failing to meet the actual needs of education administrators and teachers. Summary of the Invention

[0005] In order to achieve accurate quantitative analysis of students' classroom concentration and interaction quality, and automatically generate classroom analysis reports; this application provides an AI-based classroom content analysis method, system, device and storage medium.

[0006] In a first aspect, the present application provides an AI-based classroom content analysis method, comprising:

[0007] Real-time collection of classroom video and voice data;

[0008] Using the first multimodal fusion and scene recognition model built based on deep learning, the current classroom video and voice data are analyzed, video features and voice features are extracted, and the current classroom scene is identified and judged;

[0009] According to the preset classroom scene-interaction object mapping rules, determine the interactive object mapped corresponding to the current real-time recognized classroom scene;

[0010] Using a second multimodal fusion and effective interaction recognition model built based on deep learning, the current classroom video and voice data are analyzed to extract the behavioral characteristics of students under the conditions of the interactive objects mapped to the current classroom scene. This determines whether each student's current behavior has effectively interacted with the interactive objects mapped to the current classroom scene. The ratio of the effective interaction time between each student's behavior and the interactive objects mapped to the current classroom scene to the total detection time is calculated to calculate each student's classroom concentration.

[0011] Based on the interaction quality assessment model, including according to the preset interaction quality assessment standards matched with the interaction objects mapped corresponding to the current classroom scene, including at least one standard among the interaction accuracy standard, the interaction initiative standard, the interaction depth standard and the interaction emotional engagement standard, the classroom video and voice data in which each student's behavior generates effective interaction with the interaction objects mapped corresponding to the current classroom scene are analyzed and determined, and the interaction quality index of each student is obtained through weighted calculation;

[0012] According to the preset grade evaluation criteria, based on each student's classroom concentration and interaction quality index, an analysis report on the classroom concentration level and interaction quality level is automatically generated.

[0013] By adopting the above solution, multi-source data is collected and combined with deep learning models to fuse the data to ensure the accuracy of subsequent analysis; the classroom scene is determined to map and obtain the interactive objects, and the students' various behavioral characteristics are extracted to determine effective interaction and calculate concentration, thereby achieving accurate quantitative analysis of classroom concentration; the interaction quality assessment model is used to calculate the interaction quality index according to multiple standards to improve the accuracy of interaction quality assessment; according to the preset grade evaluation criteria, an analysis report on the classroom concentration level and interaction quality level is automatically generated.

[0014] Preferably, the automatically generating of an analysis report on the class concentration level and the interaction quality level according to the preset grade evaluation criteria and the class concentration level and interaction quality index of each student includes:

[0015] Setting a first influencing indicator for a preset evaluation standard of individual student concentration in class; the first influencing indicator includes: class time, grade of students in class, class of students in class, distance between students in class, physical condition of students in class, and classroom environment;

[0016] Setting a second influencing indicator for the preset evaluation criteria of the individual student interaction quality index; the second influencing indicator includes: classroom teaching subjects, classroom teaching content complexity, and classroom teaching interaction form;

[0017] Analyze the collected classroom video and voice data according to the preset numerical calculation rules of the first influence indicator and the preset numerical calculation rules of the second influence indicator to determine the corresponding influence indicator value of the first influence indicator and the corresponding influence indicator value of the second influence indicator;

[0018] According to the influence coefficient of the preset evaluation standard of the individual student's classroom concentration that is matched with the corresponding influence indicator value of the first influence indicator, the preset evaluation standard of the individual student's classroom concentration that is adjusted by multiplying the influence coefficient of the preset evaluation standard of the individual student's classroom concentration that is adjusted; according to the influence coefficient of the preset evaluation standard of the individual student's interaction quality index that is matched with the corresponding influence indicator value of the second influence indicator, the preset evaluation standard of the individual student's interaction quality index that is adjusted by multiplying the preset evaluation standard of the individual student's interaction quality index that is adjusted;

[0019] Based on the adjusted preset evaluation criteria for individual student classroom concentration, the classroom concentration level of each student is determined; based on the adjusted preset evaluation criteria for individual student interaction quality index, the interaction quality level of each student is determined.

[0020] By adopting the above solution, we consider and set influencing indicators related to classroom concentration and interaction quality index, determine the indicator values ​​based on the collected classroom video and voice data, and then adjust the evaluation criteria by matching the influence coefficient. We can more accurately determine each student's classroom concentration level and interaction quality level based on the student's classroom concentration and interaction quality index, and obtain a more accurate report.

[0021] Preferably, it also includes:

[0022] According to the interactive objects mapped to the current classroom scene identified in real time, analyze the current classroom video and voice data, and extract the preset abnormal behavior characteristics of each student interactive object under the conditions of the interactive objects mapped to the current classroom scene;

[0023] Performing a correlation analysis on the extracted student behavior characteristics and the preset abnormal behavior characteristics of the corresponding interaction object to obtain a behavior characteristic pair whose correlation coefficient is greater than a first preset correlation coefficient and a behavior characteristic pair whose correlation coefficient is less than a second preset correlation coefficient, wherein the first preset correlation coefficient is greater than the second preset correlation coefficient; reducing the weight of the behavior characteristic with a correlation coefficient greater than the first preset correlation coefficient to the behavior characteristic of the middle school student, and maintaining the weight of the behavior characteristic with a correlation coefficient less than the second preset correlation coefficient to the behavior characteristic of the middle school student;

[0024] The weight adjustment strategy of the preset interaction quality evaluation standard is matched according to the extracted preset abnormal behavior characteristics of each student interaction object. The preset abnormal behavior characteristics of different interaction objects are preset with the weight adjustment strategy of the preset interaction quality evaluation standard.

[0025] By adopting the above scheme, taking into account that students' classroom concentration and interaction quality index are affected by the interactive objects, and the abnormal behavior of the interactive objects often leads to changes in student behavior, a correlation analysis is performed on the student behavior characteristics and the preset abnormal behavior characteristics, and the weights are adjusted to achieve decoupling of behavioral characteristics and adjust the weights to reduce interference with student behavior characteristics, and obtain more accurate classroom concentration and interaction quality index.

[0026] Preferably, it also includes:

[0027] Determine whether the interactive object mapped to the current real-time recognized classroom scene is multiple objects. If it is determined to be multiple objects, extract the preset abnormal behavior characteristics of each interactive object;

[0028] The extracted behavioral characteristics of the students are correlated with the preset abnormal behavioral characteristics of each corresponding interactive object to obtain different correlation coefficients; it is determined whether the maximum correlation coefficient difference between different correlation coefficients is greater than the preset coefficient difference. If it is, the maximum correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient; otherwise, the weighted average correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient.

[0029] By adopting the above scheme, taking into account the possible existence of multiple interactive objects and the influence of each interactive object on it, a correlation analysis is performed on the student's behavioral characteristics and the preset abnormal behavioral characteristics of each interactive object, and an appropriate comparison method is adopted according to the gap between different correlation coefficients to more accurately evaluate the correlation between student behavioral characteristics and abnormal behavioral characteristics.

[0030] Preferably, it also includes:

[0031] Collect post-class feedback data from each student, including: post-class exercises or assessment data;

[0032] For each student, a third-party multimodal fusion and content recognition model based on deep learning is used to analyze classroom video and voice data, extracting video and voice features corresponding to the interactive objects mapped in the current classroom scene, and identifying and determining the knowledge content involved in the classroom and the knowledge content involved in classroom interaction.

[0033] Using the similarity comparison method, extract the post-class feedback data related to the knowledge content involved in the class and the knowledge content involved in the class interaction from the post-class feedback data of the corresponding students;

[0034] For each student, the accuracy rate of exercises or assessments involving knowledge content in class and the accuracy rate of exercises or assessments involving knowledge content in class interactions are statistically extracted from some of the after-class feedback data; the first correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class, and the first correction index is used to correct the classroom concentration of the corresponding student; the second correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class interactions, and the second correction index is used to correct the interaction quality index of the corresponding student; among them, the accuracy rate of exercises or assessments involving knowledge content in different classes is preset with a matching first correction index; the accuracy rate of exercises or assessments involving knowledge content in different class interactions is preset with a matching second correction index.

[0035] By adopting the above scheme, we can comprehensively consider the students' mastery of knowledge after class and match the correction coefficients of classroom concentration and interaction quality according to the mastery, so as to obtain more accurate students' classroom concentration and interaction quality.

[0036] Preferably, the automatically generating of an analysis report on the class concentration level and the interaction quality level according to the preset grade evaluation criteria and the class concentration level and interaction quality index of each student further includes:

[0037] According to the time sequence of the classroom videos, the classroom concentration level and interaction quality level of each student at the corresponding moment are inserted into the corresponding student position in the video for display;

[0038] Receive instructions for obtaining the situation of a specified student in a pre-specified time period, and display the classroom video obtained during the specified time period, including the specified student's classroom concentration level and interaction quality level, in an analysis report.

[0039] By adopting the above solution, we can statistically analyze the classroom concentration level and interaction quality level of specified students in a specified time period. By inserting the classroom concentration level and interaction quality level into the video, teachers and education administrators can obtain detailed information for specific time periods and students, thereby making more accurate teaching quality assessments and teaching decisions.

[0040] Preferably, it also includes:

[0041] The entire classroom video is divided into various time periods, and statistical analysis is performed to obtain the time periods with the highest density of all students' classroom concentration levels exceeding the preset classroom concentration levels, as well as the time periods with the highest density of all students' interaction quality levels exceeding the preset interaction quality levels, and the results are displayed in the analysis report.

[0042] By adopting the above solution, we can determine the time period when the concentration and interaction quality of all students in the classroom are optimal, which will help teachers and education administrators understand the time period when students' concentration and interaction quality are higher in the classroom.

[0043] In a second aspect, the present application provides an AI-based classroom content analysis system, comprising:

[0044] Classroom data collection module, used to collect classroom video and voice data in real time;

[0045] The classroom scene recognition module is used to analyze the current classroom video and voice data using the first multimodal fusion and scene recognition model built based on deep learning, extract video features and voice features, and identify and determine the current classroom scene;

[0046] The interactive object mapping module is used to determine the interactive object mapped to the current classroom scene identified in real time according to the preset classroom scene-interactive object mapping rules;

[0047] The student concentration acquisition module is used to use the second multimodal fusion and effective interaction recognition model built based on deep learning to analyze the current classroom video and voice data, extract the student's behavioral characteristics under the conditions of the interactive objects mapped to the current classroom scene, and determine whether each student's current behavior has effectively interacted with the interactive objects mapped to the current classroom scene; calculate the ratio of the effective interaction time between each student's behavior and the interactive objects mapped to the current classroom scene to the total detection time, and calculate each student's classroom concentration;

[0048] A student interaction quality acquisition module is used to analyze and determine classroom video and voice data in which each student's behavior generates effective interaction with the interaction object mapped to the current classroom scene based on an interaction quality assessment model, including preset interaction quality assessment standards that match the interaction object mapped to the current classroom scene, including at least one standard from among interaction accuracy, interaction initiative, interaction depth, and interaction emotional engagement, and to obtain each student's interaction quality index through weighted calculation;

[0049] The classroom content analysis report generation module is used to automatically generate an analysis report on the classroom concentration level and interaction quality level according to the preset grade evaluation criteria and the classroom concentration and interaction quality index of each student.

[0050] By adopting the above solution, multi-source data is collected, combined with deep learning models to identify classroom scenes, determine interaction objects, extract students' multi-faceted behavioral characteristics to judge effective interactions, calculate classroom concentration and interaction quality index, and finally automatically generate analysis reports on classroom concentration levels and interaction quality levels. This can achieve multi-dimensional analysis of the classroom and improve the accuracy and efficiency of classroom analysis.

[0051] In a third aspect, the present application provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0052] In a fourth aspect, the present application provides a computer device, which includes a memory, a processor, and a program stored and executable on the memory, and the program implements the steps of the above method when executed by the processor.

[0053] In summary, this application has the following beneficial effects:

[0054] 1. Using a model built using deep learning to analyze classroom video and voice data, the system improves the depth of understanding of classroom content. Through deep learning model analysis, the system determines classroom scenes and interaction objects, judges effective student interaction, calculates classroom concentration and interaction quality index, and automatically generates analysis reports on classroom concentration levels and interaction quality levels, improving the accuracy and efficiency of classroom analysis.

[0055] 2. Adjust the evaluation criteria by combining multiple influencing indicators to determine the indicator values. Adjust the evaluation criteria by matching the influence coefficients to more accurately determine the students' classroom concentration and interaction quality level;

[0056] 3. Adjust the behavioral feature weights and the preset interaction quality assessment criteria based on the preset abnormal behavioral features of the interaction objects, and select the appropriate correlation coefficient comparison method when multiple interaction objects are involved; at the same time, collect students' post-class feedback data and compare it with the classroom content to correct students' classroom concentration. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Flowchart of the AI-based classroom content analysis method described in a specific embodiment;

[0058] Figure 2 It is a structural diagram of the AI-based classroom content analysis system described in a specific embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0060] like Figure 1 As shown, the embodiment of the present application discloses an AI-based classroom content analysis method, the specific steps of which include:

[0061] S1. Real-time collection of classroom video and voice data.

[0062] Specifically, to avoid the large errors in data collected from classroom content using only voice conversion, a variety of data acquisition devices, including microphone arrays, sensors, and cameras, are selected to collect classroom videos and capture classroom voice data in real time. The microphone array can be composed of multiple voice signals, high-performance microphones with high replaceable characteristic sensitivity and wide-band matrix microphone response, and layout. Sensors can include infrared sensors, pressure sensors, sound sensors, etc. installed in various corners of the classroom to detect students' body temperature, movements, and other information. The camera device uses a high-resolution and wide-angle camera to record classroom videos in all directions.

[0063] S2. Use the first multimodal fusion and scene recognition model built based on deep learning to analyze the current classroom video and voice data, extract video features and voice features, and identify and judge the current classroom scene.

[0064] Specifically, considering different classroom scenarios, students' interaction objects are different, and students' behavioral characteristics are different under different interaction object conditions. In order to accurately judge students' concentration and interaction quality in class based on their behavioral characteristics, it is necessary to determine the current classroom scenario in advance.

[0065] Deep learning is chosen to construct the first multimodal fusion and scene recognition model, the input of which is the classroom video and voice data collected in real time, and the output is the classroom scene; specifically, the currently collected classroom video and voice data are input into the model for feature extraction: including using CNN to analyze the spatial-temporal features of several consecutive frames to extract video features, such as: the position of the teacher in the picture, the position of the student in the picture, the teacher's posture and expression, the student's posture and expression, the position of the teaching equipment and the display content of the teaching equipment, etc.; extracting voice features, including locating the sound source through the beamforming technology of the microphone array, combining voice activity detection to separate the teacher and student voices, inputting the RNN classifier to identify scene keywords, obtaining acoustic features such as the teacher's voice acoustic features and the student's voice acoustic features, as well as text content such as the teacher's output keywords and the student's output keywords; fusing the extracted video features and voice features for scene recognition.

[0066] Among them, the classroom video and voice data with historical annotations of the current classroom scene are used as training data to complete the model training of the first multimodal fusion and scene recognition model. In this embodiment, the classroom scenes specifically include: teacher teaching scenes, student free interaction scenes, teacher-student interaction scenes, and student self-study scenes, etc. Correspondingly, the video features and voice features extracted in different scenes have different focuses, such as: the video features, voice features, and text features in the teacher teaching scene include: the teacher occupies the center of the screen, the students face the teacher, the teacher's voice accounts for 80%, and has keywords such as "please look" and "pay attention"; the video features, voice features, and text features in the student free interaction scene include: students are grouped and clustered, the student voice accounts for 80%, and has keywords such as "I think" and "how to do"; the video features, voice features, and text features in the teacher-student interaction scene include: the teacher is in the student area, the teacher and the student look at each other, the students stand, and the mouth movements, student voice and teacher voice appear alternately; the video features, voice features, and text features in the student self-study scene include: the student lowers his head to face the textbook, etc.

[0067] S3. According to the preset classroom scene-interaction object mapping rules, determine the interactive object mapped corresponding to the current real-time recognized classroom scene.

[0068] Specifically, the classroom scene-interaction object mapping rules are pre-set, such as: in the teacher teaching scene, the interaction objects are teacher-student; in the student free interaction scene, the interaction objects are student-student; in the teacher-student interaction scene, the interaction objects are teacher-student; in the student self-study scene, the interaction objects are student-textbook / teaching equipment, etc.

[0069] According to the preset classroom scene-interaction object mapping rules, the interactive object corresponding to the currently recognized classroom scene is determined.

[0070] S4. Use the second multimodal fusion and effective interaction recognition model built based on deep learning to analyze the current classroom video and voice data, determine whether each student's behavior under the conditions of the interactive objects mapped corresponding to the current classroom scene produces effective interaction with the interactive objects mapped corresponding to the current classroom scene, and calculate the students' classroom concentration.

[0071] In order to evaluate the concentration, a behavior matching based on interactive objects is designed, with whether the student behavior produces effective interaction with the standard interactive objects of the current scene as the evaluation core, and a deep learning model is used to extract behavioral features under the corresponding mapped interactive objects, specifically including: using the second multimodal fusion and effective interaction recognition model constructed based on deep learning, the model convolutional neural network (CNN) and recurrent neural network can be used to improve the architecture that combines feature (RNN) extraction and scene recognition, such as: the second multimodal fusion model adopts a deep learning model, the model input is classroom video and voice data and the interactive objects mapped corresponding to the current classroom scene, and the output is whether each student behavior produces effective interaction with the interactive objects mapped corresponding to the current classroom scene, and the data of classroom video and voice data with effective interaction and interactive object conditions mapped corresponding to the classroom scene are used as training data for training and generation.

[0072] Among them, the second multimodal fusion and effective interaction recognition model constructed based on deep learning is used to analyze the current classroom video and voice data, and extract the behavioral characteristics of students under the conditions of the interactive objects mapped corresponding to the current classroom scene (involving interactions with the interactive objects mapped corresponding to the current classroom scene), including: facial expression characteristics, posture characteristics, action characteristics and voice characteristics, etc.; for the interactive objects mapped corresponding to different classroom scenes, the extracted student behavioral characteristics have different focuses, such as: in the teacher teaching scene, the student's facial expression characteristics focus on the proportion of time looking at the teacher, the activeness of nodding / smiling / doubtful expressions, etc., the posture characteristics focus on the correctness of sitting facing the teacher, the head tilt angle, etc., the action characteristics focus on turning books / taking notes, etc., and the voice features include: the proportion of voice time of following the teacher's instructions; for example: in the student free interaction scene, the student's facial expression characteristics focus on the proportion of time looking at peers, the activeness of nodding / smiling / doubtful expressions, etc., the posture characteristics focus on the correctness of sitting facing the discussion object, etc., the action characteristics focus on transmitting information, etc., and the voice features include: the length of speech and the matching degree of speech content and discussion topic, etc.

[0073] Therefore, the second multimodal fusion and effective interaction recognition model constructed based on deep learning is used to analyze the current classroom video and voice data to determine whether each student's behavior under the conditions of the interactive objects mapped corresponding to the current classroom scene produces effective interaction with the interactive objects mapped corresponding to the current classroom scene.

[0074] The ratio of the effective interaction time between each student's behavior and the interactive object mapped to the current classroom scene to the total detection time is counted to calculate each student's classroom concentration; the specific formula includes:

[0075]

[0076] Among them, since the test starts from the class, the total test time refers to the current class progress time.

[0077] S5. Calculate and obtain the interaction quality index of each student based on the interaction quality evaluation model.

[0078] Specifically, an interaction quality assessment model is pre-set; taking into account the differences in interactions between different classroom scenarios and different interaction objects, different interaction quality assessment standards are set for different classroom scenarios and different interaction scenarios, including preset interaction quality assessment standards that match the interaction objects mapped corresponding to the current classroom scenario, including at least one standard among the interaction accuracy standard, interaction initiative standard, interaction depth standard and interaction emotional investment standard; among which, the evaluation indicator of the interaction accuracy standard is content relevance, and the corresponding calculation method includes: the similarity between keywords in voice features and the current classroom topic, etc.; the evaluation indicator of the interaction initiative standard is the number of interactions, and the corresponding calculation method includes: the number of students actively asking questions / speaking, etc.; the evaluation indicator of the interaction depth is the proportion of multiple rounds of dialogues, and the corresponding calculation method includes: the proportion of logically related dialogues of more than N consecutive rounds, etc.; the evaluation indicator of the interaction emotional investment standard is language emotional acuteness, and the corresponding calculation method includes: calculating the proportion of positive emotions through an emotion classification model, etc.

[0079] In this embodiment, the interactive objects mapped corresponding to the current classroom scene are teachers and students or students and students, and the preset interaction quality evaluation standards include: interaction accuracy standard, interaction initiative standard, interaction depth standard, and interaction emotional involvement standard, etc.; the interactive objects mapped corresponding to the current classroom scene are students and teaching materials / teaching equipment, and the preset interaction quality evaluation standards include: interaction initiative standard, interaction emotional involvement standard, etc.

[0080] Therefore, according to the preset interaction quality evaluation standard that matches the interactive objects mapped to the current classroom scene, the classroom video and voice data in which each student behavior effectively interacts with the interactive objects mapped to the current classroom scene are analyzed and determined. The specific interaction quality index calculation formula is:

[0081]

[0082] in, as well as The sum is 1, and for different preset interaction quality evaluation criteria, as well as The corresponding value is 0.

[0083] S6. According to the preset grade evaluation criteria, based on each student's classroom concentration and interaction quality index, an analysis report on the classroom concentration level and interaction quality level is automatically generated.

[0084] Specifically, the preset grade evaluation criteria include the evaluation criteria for classroom concentration and the evaluation criteria for interaction quality. The evaluation criteria for classroom concentration are used to grade the classroom concentration of each student, and the evaluation criteria for interaction quality grade are used to grade the interaction quality of each student, and an analysis report on the classroom concentration grade and interaction quality grade is automatically generated.

[0085] In a specific embodiment, considering the factors that affect each student's classroom concentration and interaction quality index in the classroom, in order to more accurately determine the classroom concentration level and interaction quality level of different students, the preset grade evaluation criteria are adjusted. The method automatically generates an analysis report on the classroom concentration level and interaction quality level according to the preset grade evaluation criteria and the classroom concentration and interaction quality index of each student, including:

[0086] Set the first influencing indicator of the preset evaluation criteria for a single student's classroom concentration; the first influencing indicator includes: class time, the grade of students in the classroom, the class of students in the classroom, the distance between students in the classroom, the physical condition of students in the classroom, and the classroom environment; set the corresponding preset numerical calculation rules for a single first influencing indicator, such as: students' classroom concentration is higher during the period from 9 to 10 o'clock in the day, the corresponding preset numerical value is higher, and the corresponding preset numerical value is matched for each period of the 24 hours in a day when the classroom time is correspondingly; correspondingly, the younger the grade of the students in the classroom or the class they are in is Class B, the lower the student's classroom concentration is, the corresponding preset numerical value is lower, and the corresponding preset numerical value is matched for the size of the grade of the students in the classroom and the size of the students in the classroom. The corresponding preset numerical value is matched for Class A or Class B in the class; accordingly, the farther the students are from the blackboard in the classroom, the lower the student's classroom concentration, the lower the corresponding preset numerical value, and the corresponding preset numerical value is matched for the distance position of the students in the classroom; accordingly, the worse the physical condition of the students in the classroom (determined by judging the body temperature through infrared cameras, judging whether the expression is painful by the camera device, etc.), the lower the student's classroom concentration, the lower the corresponding preset numerical value, and the corresponding preset numerical value is matched for the physical condition of the students in the classroom; accordingly, the more environmental noise in the classroom (determined by extracting classroom noise characteristics), the lower the student's classroom concentration, the lower the corresponding preset numerical value, and the corresponding preset numerical value is matched for the environmental noise in the classroom.

[0087] A second influencing indicator is set for the preset evaluation criteria of a single student interaction quality index; the second influencing indicator includes: classroom teaching subjects, classroom teaching content complexity, classroom teaching interaction form, etc.; corresponding preset numerical calculation rules are set for a single second influencing indicator, such as: for the main teaching subjects, the higher the student interaction quality, the higher the corresponding interaction quality index, and the corresponding preset numerical matching setting is made for whether the classroom teaching subject is the main subject; the more complex the complexity of the classroom teaching content (the current classroom teaching content complexity can be determined according to the accuracy of the course teaching content in the curriculum), the lower the student interaction quality, the lower the corresponding interaction quality index, and the corresponding preset numerical matching setting is made for the complexity of the classroom teaching content; for the interaction form of remote classroom teaching, the lower the student interaction quality, the lower the corresponding interaction quality index, and the corresponding preset numerical matching setting is made for whether the classroom teaching interaction form is remote teaching.

[0088] According to the preset numerical calculation rules for the first influencing indicator and the preset numerical calculation rules for the second influencing indicator, the collected classroom video and voice data are analyzed to determine the corresponding influence indicator value of the first influencing indicator and the corresponding influence indicator value of the second influencing indicator; wherein, the corresponding influence indicator value of the first influencing indicator is obtained by weighted calculation of the corresponding influence indicator value of a single first influencing indicator, and the corresponding influence indicator value of the second influencing indicator is obtained by weighted calculation of the corresponding influence indicator value of a single second influencing indicator.

[0089] According to the influence coefficient of the preset evaluation standard of the classroom concentration of a single student matched with the corresponding influence indicator value of the first influence indicator, the preset evaluation standard of the classroom concentration of a single student is adjusted by multiplying the influence coefficient of the preset evaluation standard of the classroom concentration of a single student; wherein, the corresponding influence indicator values ​​of different first influence indicators are all set with the influence coefficient of the preset evaluation standard of the classroom concentration that matches them; according to the influence coefficient of the preset evaluation standard of the interaction quality index that matches the corresponding influence indicator value of the second influence indicator, the preset evaluation standard of the interaction quality index of a single student is adjusted by multiplying the influence coefficient of the preset evaluation standard of the interaction quality index of a single student, and the corresponding influence indicator values ​​of different second influence indicators are all set with the influence coefficient of the preset evaluation standard of the interaction quality index that matches them.

[0090] Based on the adjusted preset evaluation criteria for individual student classroom concentration, the classroom concentration level of each student is determined; based on the adjusted preset evaluation criteria for individual student interaction quality index, the interaction quality level of each student is determined.

[0091] In a specific embodiment, considering that students' classroom concentration and interaction quality are often closely related to the interaction object, if the interaction object has abnormal behavior, it will correspondingly affect the students' classroom concentration and interaction quality. In order to further improve the accuracy of the analysis of students' classroom concentration and interaction quality, the method further includes:

[0092] According to the interactive objects mapped to the current classroom scene identified in real time, the current classroom video and voice data are analyzed to extract the preset abnormal behavior characteristics of each student interactive object under the conditions of the interactive objects mapped to the current classroom scene; for example, when the interactive object is a teacher, the preset abnormal behavior characteristics include: abnormal speaking speed, voice interruption, anxious or tired facial expression, abnormal movement, blackboard writing errors, and other abnormal behavior characteristics of teachers that represent teacher fatigue; when the interactive object is a student, the preset abnormal behavior characteristics include: abnormal student movement, abnormal student voice intensity, and other abnormal student behavior characteristics that represent the restlessness of the student group; when the interactive object is a teaching material / teaching equipment, the preset abnormal behavior characteristics include: abnormal projection screen, blackboard content obstruction, device screen interruption, and other abnormal behavior characteristics that represent teaching material content obstruction / teaching equipment failure;

[0093] A correlation analysis is performed on the extracted student behavior characteristics and the preset abnormal behavior characteristics of the corresponding interactive object. In this embodiment, the correlation analysis is calculated using the Pearson correlation coefficient to obtain behavior feature pairs with correlation coefficients greater than the first preset correlation coefficient and behavior feature pairs with correlation coefficients less than the second preset correlation coefficient, where the first preset correlation coefficient is greater than the second preset correlation coefficient; the weight of the behavior features with correlation coefficients greater than the first preset correlation coefficient for the middle school student behavior characteristics is reduced, such as: if the correlation between the teacher's abnormal speaking speed and the student's frowning facial expression is greater than the first preset correlation coefficient, it indicates that the current student's facial representation is an interference item caused by the teacher's abnormal behavior, and the weight of the facial expression feature in the concentration judgment is reduced; for example: if the correlation between the occlusion of the blackboard content and the student's line of sight deviation is greater than the first preset correlation coefficient, it indicates that the current student's inattention is due to the occlusion of the blackboard, and the weight of the line of sight deviation facial expression feature in the concentration judgment is reduced; the weight of the behavior features with correlation coefficients less than the second preset correlation coefficient for the middle school student behavior characteristics is maintained, and the behavior feature pairs with correlation coefficients less than the second preset correlation coefficient indicate that the student's concentration is not shifted due to the abnormality of the interactive object, so the weight remains unchanged.

[0094] The weight adjustment strategy of the preset interaction quality evaluation standard is matched according to the preset abnormal behavior characteristics extracted for each student interaction object. The preset abnormal behavior characteristics of different interaction objects are preset with corresponding weight adjustment strategies of the preset interaction quality evaluation standard, specifically including: the abnormal behavior characteristics of teachers that represent teacher fatigue correspond to the weights of reducing the interaction initiative standard and the interaction depth standard; the abnormal behavior characteristics of students that represent the restlessness of the student group correspond to the weights of reducing the interaction depth standard; the abnormal behavior characteristics that represent the obstruction of textbook content / teaching equipment failure correspond to the weights of reducing the interaction accuracy standard and the interaction emotional investment.

[0095] In addition, considering that there may be multiple objects for user interaction, such as multiple students interacting, in order to avoid the simultaneous impact of multiple interactive objects on students' classroom concentration, the method also includes: determining whether the interactive objects mapped corresponding to the current real-time identified classroom scene are multiple objects; if it is determined to be multiple interactive objects, extracting the preset abnormal behavior characteristics of each interactive object; performing correlation analysis on the extracted student behavior characteristics and the preset abnormal behavior characteristics of each corresponding interactive object to obtain different correlation coefficients; judging whether the maximum correlation coefficient difference between different correlation coefficients is greater than the preset coefficient difference; if it is greater, selecting the maximum correlation coefficient to be compared with the first preset correlation coefficient and the second preset correlation coefficient; otherwise, selecting the weighted average correlation coefficient to be compared with the first preset correlation coefficient and the second preset correlation coefficient.

[0096] In a specific embodiment, considering the problems in judging the concentration and interaction quality in class, the method further includes:

[0097] Collect after-class feedback data for each student, including after-class exercises or assessment data.

[0098] For each student, a third multimodal fusion and content recognition model constructed based on deep learning is used to analyze the current classroom video and voice data, extract the video features and voice features under the conditions of the interactive objects mapped corresponding to the current classroom scene, and identify and judge the classroom-related knowledge content and classroom interaction-related knowledge content; the third multimodal fusion and content recognition model adopts a deep learning model, the model input is the classroom video and voice data and the interactive objects mapped corresponding to the current classroom scene, and the output is the classroom-related knowledge content and classroom interaction-related knowledge content, and is generated through the historical interactive objects mapped corresponding to the classroom scenes of each student, classroom video and voice data, and the classroom-related knowledge content and classroom interaction-related knowledge content in the classroom video and voice data as training data.

[0099] Using the similarity comparison method, extract the post-class feedback data related to the knowledge content involved in the class and the knowledge content involved in the class interaction from the post-class feedback data of the corresponding students;

[0100] For each student, the accuracy rate of exercises or assessments involving knowledge content in class and the accuracy rate of exercises or assessments involving knowledge content in class interactions are statistically extracted from some of the after-class feedback data; the first correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class, and the first correction index is used to correct the classroom concentration of the corresponding student; the second correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class interactions, and the second correction index is used to correct the interaction quality index of the corresponding student; among them, the accuracy rate of exercises or assessments involving knowledge content in different classes is preset with a matching first correction index; the accuracy rate of exercises or assessments involving knowledge content in different class interactions is preset with a matching second correction index.

[0101] In a specific embodiment, in order to better assist teachers in understanding each student's classroom concentration and interaction quality, the method automatically generates an analysis report on the classroom concentration level and interaction quality level according to the preset grade evaluation criteria and the classroom concentration and interaction quality index of each student, further comprising:

[0102] According to the time sequence of the classroom videos, the classroom concentration level and interaction quality level of each student at the corresponding moment are inserted into the corresponding student position in the video for display;

[0103] In order to facilitate teachers to check the classroom concentration and interaction quality for specific time periods (such as the teaching period of important knowledge points designed in the lesson plan) and specific students (such as students whose grades have fluctuated in recent periods), instructions for obtaining the situation of specified students in pre-specified time periods are received, and the classroom videos obtained for the specified time periods, including the classroom concentration level and interaction quality level of the specified students, are displayed in the analysis report.

[0104] In addition, based on the needs of analyzing the quality of classroom teaching as a whole, the method also includes:

[0105] The entire classroom video is divided into various time periods, and statistical analysis is performed to obtain the time periods with the highest density of all students' classroom concentration levels exceeding the preset classroom concentration levels, as well as the time periods with the highest density of all students' interaction quality levels exceeding the preset interaction quality levels, and the results are displayed in the analysis report.

[0106] like Figure 2 As shown, the embodiment of the present application discloses an AI-based classroom content analysis system, which specifically includes:

[0107] Classroom data collection module 101, used to collect classroom video and voice data in real time;

[0108] The classroom scene recognition module 102 is used to analyze the current classroom video and voice data using a first multimodal fusion and scene recognition model built based on deep learning, extract video features and voice features, and recognize and determine the current classroom scene;

[0109] The interactive object mapping module 103 is used to determine the interactive object mapped to the current classroom scene recognized in real time according to the preset classroom scene-interactive object mapping rules;

[0110] The student concentration acquisition module 104 is used to use the second multimodal fusion and effective interaction recognition model constructed based on deep learning to analyze the current classroom video and voice data, extract the student's behavioral characteristics under the conditions of the interactive objects mapped to the current classroom scene, and determine whether each student's current behavior has effectively interacted with the interactive objects mapped to the current classroom scene; calculate the ratio of the effective interaction time between each student's behavior and the interactive objects mapped to the current classroom scene to the total detection time, and calculate the classroom concentration of each student;

[0111] The student interaction quality acquisition module 105 is configured to analyze classroom video and voice data to determine whether each student's behavior generates effective interaction with the interaction object mapped to the current classroom scene based on an interaction quality assessment model, including preset interaction quality assessment criteria that match the interaction object mapped to the current classroom scene, including at least one of the following criteria: interaction accuracy, interaction initiative, interaction depth, and interaction emotional engagement. The module then performs weighted calculation to obtain an interaction quality index for each student.

[0112] The classroom content analysis report generation module 106 is used to automatically generate an analysis report on the classroom concentration level and interaction quality level according to the preset level evaluation criteria and the classroom concentration and interaction quality index of each student.

[0113] In a specific embodiment, the system further includes: a student concentration acquisition optimization module 107, which is used to analyze the current classroom video and voice data according to the interactive objects mapped to the current classroom scene identified in real time, and extract the preset abnormal behavior characteristics of each student interactive object under the conditions of the interactive objects mapped to the current classroom scene; perform correlation analysis on the extracted student behavior characteristics and the preset abnormal behavior characteristics of the corresponding interactive objects, and obtain behavior feature pairs with correlation coefficients greater than a first preset correlation coefficient and behavior feature pairs with correlation coefficients less than a second preset correlation coefficient, the first preset correlation coefficient being greater than the second preset correlation coefficient; reduce the weight of the behavior characteristics with correlation coefficients greater than the first preset correlation coefficient to the middle school student behavior characteristics, and maintain the weight of the behavior characteristics with correlation coefficients less than the second preset correlation coefficient to the middle school student behavior characteristics;

[0114] It is also used to determine whether the interactive objects mapped corresponding to the classroom scene currently identified in real time are multiple objects; if it is determined to be multiple objects, the preset abnormal behavior characteristics of each interactive object are extracted; the extracted student behavior characteristics are correlated with the preset abnormal behavior characteristics of each corresponding interactive object to obtain different correlation coefficients; it is judged whether the maximum correlation coefficient difference between different correlation coefficients is greater than the preset coefficient difference. If it is, the maximum correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient; otherwise, the weighted average correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient.

[0115] The student interaction quality acquisition optimization module 108 is used to match the weight adjustment strategy of the preset interaction quality evaluation standard according to the preset abnormal behavior characteristics of each student interaction object extracted. The preset abnormal behavior characteristics of different interaction objects are preset with the weight adjustment strategy of the preset interaction quality evaluation standard that matches them.

[0116] In a specific embodiment, the system further includes: a student concentration and interaction quality correction module 109, which is used to collect after-class feedback data of each student, including: after-class exercise or assessment data; for each student, using a third multimodal fusion and content recognition model constructed based on deep learning, analyzing classroom video and voice data, extracting video features and voice features under the interactive object conditions mapped corresponding to the current classroom scene, identifying and judging the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction; using a similarity comparison method, extracting part of the after-class feedback data related to the knowledge content involved in the classroom and the knowledge content involved in the classroom interaction from the corresponding student's after-class feedback data; for each For students, the accuracy rate of exercises or assessments involving knowledge content in class and the accuracy rate of exercises or assessments involving knowledge content in class interaction are statistically extracted from some post-class feedback data; the first correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class, and the first correction index is used to correct the classroom concentration of the corresponding students; the second correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class interaction, and the second correction index is used to correct the interaction quality index of the corresponding students; among them, the accuracy rate of exercises or assessments involving knowledge content in different classes is preset with a matching first correction index; the accuracy rate of exercises or assessments involving knowledge content in different class interactions is preset with a matching second correction index.

[0117] The embodiment of the present application also discloses a computer-readable storage medium.

[0118] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above-mentioned AI-based classroom content analysis method. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0119] The embodiment of the present application also discloses a computer device.

[0120] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned AI-based classroom content analysis method.

[0121] The above are all preferred embodiments of the present application and are not intended to limit the scope of protection of this application. Unless otherwise stated, any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features. In other words, unless otherwise stated, each feature is merely an example of a series of equivalent or similar features.

Claims

1. A classroom content analysis method based on AI, characterized in that: include: Real-time collection of classroom video and voice data; Using the first multimodal fusion and scene recognition model built based on deep learning, the current classroom video and voice data are analyzed, video features and voice features are extracted, and the current classroom scene is identified and judged; According to the preset classroom scene-interaction object mapping rules, determine the interactive object mapped corresponding to the current real-time recognized classroom scene; Using a second multimodal fusion and effective interaction recognition model built based on deep learning, the current classroom video and voice data are analyzed to extract the behavioral characteristics of students under the conditions of the interactive objects mapped to the current classroom scene, and to determine whether each student's current behavior has effectively interacted with the interactive objects mapped to the current classroom scene; Count the ratio of the effective interaction time between each student's behavior and the interactive object mapped to the current classroom scene to the total detection time, and calculate each student's classroom concentration; Based on the interaction quality assessment model, including according to the preset interaction quality assessment standards matched with the interaction objects mapped corresponding to the current classroom scene, including at least one standard among the interaction accuracy standard, the interaction initiative standard, the interaction depth standard and the interaction emotional engagement standard, the classroom video and voice data in which each student's behavior generates effective interaction with the interaction objects mapped corresponding to the current classroom scene are analyzed and determined, and the interaction quality index of each student is obtained through weighted calculation; According to the preset grade evaluation criteria, based on each student's class concentration and interaction quality index, an analysis report on the class concentration level and interaction quality level is automatically generated; Also includes: According to the interactive objects mapped to the current classroom scene identified in real time, analyze the current classroom video and voice data, and extract the preset abnormal behavior characteristics of each student interactive object under the conditions of the interactive objects mapped to the current classroom scene; Performing a correlation analysis on the extracted student behavior characteristics and the preset abnormal behavior characteristics of the corresponding interaction object to obtain a behavior characteristic pair whose correlation coefficient is greater than a first preset correlation coefficient and a behavior characteristic pair whose correlation coefficient is less than a second preset correlation coefficient, wherein the first preset correlation coefficient is greater than the second preset correlation coefficient; reducing the weight of the behavior characteristic with a correlation coefficient greater than the first preset correlation coefficient to the behavior characteristic of the middle school student, and maintaining the weight of the behavior characteristic with a correlation coefficient less than the second preset correlation coefficient to the behavior characteristic of the middle school student; According to the extracted preset abnormal behavior characteristics of each student interaction object, a weight adjustment strategy for matching the preset interaction quality evaluation standard is used. The preset abnormal behavior characteristics of different interaction objects are preset with corresponding weight adjustment strategies for the preset interaction quality evaluation standard; Also includes: Determine whether the interactive object mapped to the current real-time recognized classroom scene is multiple objects. If it is determined to be multiple objects, extract the preset abnormal behavior characteristics of each interactive object; Perform correlation analysis on the extracted student behavior characteristics and the preset abnormal behavior characteristics of each corresponding interaction object to obtain different correlation coefficients; determine whether the maximum correlation coefficient difference between different correlation coefficients is greater than the preset coefficient difference; if greater, choose to use the maximum correlation coefficient to compare with the first preset correlation coefficient and the second preset correlation coefficient; otherwise, choose to use the weighted average correlation coefficient to compare with the first preset correlation coefficient and the second preset correlation coefficient.

2. The AI-based classroom content analysis method according to claim 1, characterized in that: The analysis report on the class concentration level and the interaction quality level is automatically generated according to the preset grade evaluation criteria and the class concentration level and interaction quality index of each student, including: Setting a first influencing indicator for a preset evaluation standard of individual student concentration in class; the first influencing indicator includes: class time, grade of students in class, class of students in class, distance between students in class, physical condition of students in class, and classroom environment; Setting a second influencing indicator for the preset evaluation criteria of the individual student interaction quality index; the second influencing indicator includes: classroom teaching subjects, classroom teaching content complexity, and classroom teaching interaction form; Analyze the collected classroom video and voice data according to the preset numerical calculation rules of the first influence indicator and the preset numerical calculation rules of the second influence indicator to determine the corresponding influence indicator value of the first influence indicator and the corresponding influence indicator value of the second influence indicator; According to the influence coefficient of the preset evaluation standard of the individual student's classroom concentration that is matched with the corresponding influence indicator value of the first influence indicator, the preset evaluation standard of the individual student's classroom concentration that is adjusted by multiplying the influence coefficient of the preset evaluation standard of the individual student's classroom concentration that is adjusted; according to the influence coefficient of the preset evaluation standard of the individual student's interaction quality index that is matched with the corresponding influence indicator value of the second influence indicator, the preset evaluation standard of the individual student's interaction quality index that is adjusted by multiplying the preset evaluation standard of the individual student's interaction quality index that is adjusted; Based on the adjusted preset evaluation criteria for individual student classroom concentration, the classroom concentration level of each student is determined; based on the adjusted preset evaluation criteria for individual student interaction quality index, the interaction quality level of each student is determined.

3. The AI-based classroom content analysis method according to claim 1, characterized in that: Also includes: Collect post-class feedback data from each student, including: post-class exercises or assessment data; For each student, a third-party multimodal fusion and content recognition model based on deep learning is used to analyze classroom video and voice data, extracting video and voice features corresponding to the interactive objects mapped in the current classroom scene, and identifying and determining the knowledge content involved in the classroom and the knowledge content involved in classroom interaction. Using the similarity comparison method, extract the post-class feedback data related to the knowledge content involved in the class and the knowledge content involved in the class interaction from the post-class feedback data of the corresponding students; For each student, the accuracy rate of exercises or assessments involving knowledge content in class and the accuracy rate of exercises or assessments involving knowledge content in class interactions are statistically extracted from some of the after-class feedback data; the first correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class, and the first correction index is used to correct the classroom concentration of the corresponding student; the second correction index is matched according to the accuracy rate of exercises or assessments involving knowledge content in class interactions, and the second correction index is used to correct the interaction quality index of the corresponding student; among them, the accuracy rate of exercises or assessments involving knowledge content in different classes is preset with a matching first correction index; the accuracy rate of exercises or assessments involving knowledge content in different class interactions is preset with a matching second correction index.

4. The AI-based classroom content analysis method according to claim 2, characterized in that: The automatic generation of an analysis report on the class concentration level and interaction quality level according to the preset grade evaluation criteria and the class concentration level and interaction quality index of each student also includes: According to the time sequence of the classroom videos, the classroom concentration level and interaction quality level of each student at the corresponding moment are inserted into the corresponding student position in the video for display; Receive instructions for obtaining the situation of a specified student in a pre-specified time period, and display the classroom video obtained during the specified time period, including the specified student's classroom concentration level and interaction quality level, in an analysis report.

5. The AI-based classroom content analysis method according to claim 2, characterized in that: Also includes: The entire classroom video is divided into various time periods, and statistical analysis is performed to obtain the time periods with the highest density of all students' classroom concentration levels exceeding the preset classroom concentration levels, as well as the time periods with the highest density of all students' interaction quality levels exceeding the preset interaction quality levels, and the results are displayed in the analysis report.

6. An AI-based classroom content analysis system, characterized in that: include: Classroom data collection module, used to collect classroom video and voice data in real time; The classroom scene recognition module is used to analyze the current classroom video and voice data using the first multimodal fusion and scene recognition model built based on deep learning, extract video features and voice features, and identify and determine the current classroom scene; The interactive object mapping module is used to determine the interactive object mapped to the current classroom scene identified in real time according to the preset classroom scene-interactive object mapping rules; The student concentration acquisition module is used to use the second multimodal fusion and effective interaction recognition model built based on deep learning to analyze the current classroom video and voice data, extract the student's behavioral characteristics under the conditions of the interactive objects mapped to the current classroom scene, and determine whether each student's current behavior has effectively interacted with the interactive objects mapped to the current classroom scene; calculate the ratio of the effective interaction time between each student's behavior and the interactive objects mapped to the current classroom scene to the total detection time, and calculate each student's classroom concentration; A student interaction quality acquisition module is used to analyze and determine classroom video and voice data in which each student's behavior generates effective interaction with the interaction object mapped to the current classroom scene based on an interaction quality assessment model, including preset interaction quality assessment standards that match the interaction object mapped to the current classroom scene, including at least one standard from among interaction accuracy, interaction initiative, interaction depth, and interaction emotional engagement, and to obtain each student's interaction quality index through weighted calculation; The classroom content analysis report generation module is used to automatically generate an analysis report on the classroom concentration level and interaction quality level based on the preset grade evaluation criteria and the classroom concentration and interaction quality index of each student; The student concentration acquisition optimization module is used to analyze the current classroom video and voice data based on the interactive objects mapped to the current classroom scene in real time, and extract the preset abnormal behavior characteristics of each student interactive object under the conditions of the interactive objects mapped to the current classroom scene; Performing a correlation analysis on the extracted student behavior characteristics and the preset abnormal behavior characteristics of the corresponding interaction object to obtain a behavior characteristic pair whose correlation coefficient is greater than a first preset correlation coefficient and a behavior characteristic pair whose correlation coefficient is less than a second preset correlation coefficient, wherein the first preset correlation coefficient is greater than the second preset correlation coefficient; reducing the weight of the behavior characteristic with a correlation coefficient greater than the first preset correlation coefficient to the behavior characteristic of the middle school student, and maintaining the weight of the behavior characteristic with a correlation coefficient less than the second preset correlation coefficient to the behavior characteristic of the middle school student; It is also used to determine whether the interactive object mapped to the current real-time recognized classroom scene is multiple objects; if it is determined to be multiple objects, extract the preset abnormal behavior characteristics of each interactive object; Correlation analysis is performed on the extracted student behavior characteristics and the preset abnormal behavior characteristics of each corresponding interaction object to obtain different correlation coefficients; it is determined whether the maximum correlation coefficient difference between the different correlation coefficients is greater than the preset coefficient difference; if so, the maximum correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient; Otherwise, the weighted average correlation coefficient is selected for comparison with the first preset correlation coefficient and the second preset correlation coefficient; The student interaction quality acquisition optimization module is used to match the weight adjustment strategy of the preset interaction quality evaluation standard based on the preset abnormal behavior characteristics of each student interaction object extracted. The preset abnormal behavior characteristics of different interaction objects are preset with the weight adjustment strategy of the preset interaction quality evaluation standard that matches them.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 5.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a program stored and executable on the memory, and when the program is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images