AI classroom behavior collaborative analysis method and system

Abstract

The invention discloses an AI classroom behavior collaborative analysis method and system. The method comprises the following steps: acquiring classroom environment data in real time; preprocessing the classroom environment data to respectively obtain target student video data, target student audio data, target student action data and target student expression data; performing feature extraction on the target student video data, the target student audio data, the target student action data and the target student expression data to respectively obtain expression features, voice content features, action features and posture features; performing comprehensive processing on the expression features, the voice content features, the action features and the posture features to obtain comprehensive features; performing multiple recognition on the comprehensive features, and performing post-processing on an initial recognition result obtained by recognition to obtain a target recognition result; and performing quantitative evaluation on the target identification result to generate a behavior analysis report. According to the invention, behaviors of students can be accurately identified and evaluated in real time, and classroom management and teaching optimization levels are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of trajectory tracking technology, and in particular to an AI classroom behavior collaborative analysis method and system. Background Art

[0002] Traditional classroom management and behavior analysis methods have many problems, which limit the real-time and accuracy of classroom behavior analysis and make it difficult to meet the needs of modern education for refined management and personalized teaching.

[0003] First, in terms of data collection, traditional classroom management and behavior analysis methods mainly rely on observation and manual recording by teachers or researchers, which is not only time-consuming and labor-intensive, but also difficult to be comprehensive and objective. Even when using cameras, microphones and other devices for data collection, data synchronization and integration is still a problem. The collection of multi-source data requires a high degree of coordination between various devices, but due to the wide variety of devices and different sampling frequencies, it often leads to inconsistencies in data in time and space, affecting subsequent analysis and processing.

[0004] Secondly, in terms of data processing, traditional classroom management and behavior analysis methods often rely on post-analysis, which cannot provide real-time feedback and cannot adjust teaching strategies in a timely manner. Although existing video and audio-based data processing technologies can recognize students' expressions, voice content, and actions to a certain extent, they have significant processing delays due to the high complexity of the algorithm and the large demand for computing resources.

[0005] Furthermore, in terms of behavior recognition and assessment, traditional classroom management and behavior analysis methods often use simple statistical analysis or rule-based algorithms, which are difficult to cope with complex and changing classroom environments. Although existing behavior recognition technologies can analyze data through machine learning and deep learning algorithms, due to the lack of comprehensive use of multimodal data, they can often only analyze from a single dimension and cannot fully evaluate students' attention, participation, and interaction. Summary of the invention

[0006] The purpose of the present invention is to collect and process data in real time and accurately, and to achieve real-time identification and effective evaluation of student behavior. In order to achieve the above purpose, the present invention provides an AI classroom behavior collaborative analysis method and system.

[0007] In a first aspect, an embodiment of the present invention provides an AI classroom behavior collaborative analysis method, comprising:

[0008] Collecting classroom environment data in real time, wherein the classroom environment data includes initial student video data, initial student audio data, initial student action data and initial student expression data;

[0009] Preprocessing the classroom environment data to obtain target student video data, target student audio data, target student action data, and target student expression data;

[0010] Extracting features from the target student video data, the target student audio data, the target student action data, and the target student expression data to obtain expression features, speech content features, action features, and posture features, respectively;

[0011] Comprehensively processing the expression features, speech content features, action features and posture features to obtain comprehensive features, wherein the comprehensive processing includes single-modal attention mechanism processing, feature fusion and multi-modal attention mechanism processing;

[0012] Perform multiple recognition on the comprehensive features, and post-process the initial recognition results obtained to obtain target recognition results, wherein the multiple recognitions include attention level recognition, participation recognition, and interaction status recognition, and the target recognition results include the target student's attention level, the target student's participation, and the target student's interaction status;

[0013] The target recognition result is quantitatively evaluated to generate a behavior analysis report, wherein the behavior analysis report includes a quantitative value of the student's attention level, a quantitative value of the student's participation level, and a quantitative value of the student's interaction status.

[0014] Preferably, after the target recognition result is quantitatively evaluated and a behavior analysis report is generated, the method further includes:

[0015] Real-time feedback and teaching suggestions are provided based on the behavior analysis report.

[0016] Preferably, the preprocessing of the classroom environment data to obtain target student video data, target student audio data, target student action data and target student expression data respectively includes:

[0017] Performing time synchronization on the initial student video data, the initial student audio data, the initial student action data and the initial student expression data to obtain intermediate student video data, intermediate student audio data, intermediate student action data and intermediate student expression data;

[0018] The intermediate student video data, intermediate student audio data, intermediate student action data and intermediate student expression data are subjected to denoising and filtering processing to obtain target student video data, target student audio data, target student action data and target student expression data respectively.

[0019] Preferably, the feature extraction of the target student video data, the target student audio data, the target student action data and the target student expression data to obtain expression features, speech content features, action features and posture features respectively includes:

[0020] Performing a first feature extraction on the target student video data and the target student expression data to obtain expression features and posture features, wherein the first feature extraction includes attention mechanism processing, graph convolution operation and feature fusion;

[0021] Performing a second feature extraction on the target student audio data to obtain speech content features, wherein the second feature extraction includes global feature extraction, key feature extraction and self-supervised learning;

[0022] Performing a third feature extraction on the target student's motion data to obtain motion features, wherein the third feature extraction includes spatiotemporal feature extraction, time series feature extraction and motion classification.

[0023] Preferably, the second feature extraction is performed on the target student audio data to obtain the speech content feature, including:

[0024] Performing global feature extraction on the target student audio data based on the Transformer network to obtain global features of the speech content;

[0025] Extract key features of the global features of the speech content based on an attention mechanism to obtain key features of the speech content;

[0026] Self-supervised learning is performed on the key features of the speech content to obtain speech content features.

[0027] Preferably, the expression features, voice content features, action features and posture features are comprehensively processed to obtain comprehensive features, including:

[0028] The expression features, speech content features, action features and posture features are processed by a unimodal attention mechanism to obtain weighted expression features, weighted speech content features, weighted action features and weighted posture features;

[0029] Performing feature fusion on the weighted expression feature, weighted speech content feature, weighted action feature and weighted posture feature to obtain a fusion feature;

[0030] The fused features are processed by a multimodal attention mechanism to obtain comprehensive features.

[0031] Preferably, a first preset formula is used to characterize the unimodal attention mechanism processing, and the first preset formula is as follows:

[0032]

[0033] Among them, F att represents weighted expression feature or weighted speech content feature or weighted action feature or weighted posture feature, K represents the total number of expression feature points or the total number of speech content feature points or the total number of action feature points or the total number of posture feature points, α k represents the attention weight of the kth expression feature point, the attention weight of the kth speech content feature point, the attention weight of the kth action feature point, or the attention weight of the kth posture feature point, f k represents the kth expression feature point or the kth speech content feature point or the kth action feature point or the kth posture feature point, and F represents the expression feature or the speech content feature or the action feature or the posture feature;

[0034] A second preset formula is used to characterize the feature fusion, and the second preset formula is as follows:

[0035]

[0036] Among them, F fusion represents the fusion feature, represents the feature concatenation operation, represents the weighted posture feature, represents the weighted speech content features, represents the weighted action feature, represents weighted expression features;

[0037] The multimodal attention mechanism processing is characterized by a third preset formula, and the third preset formula is as follows:

[0038]

[0039] Among them, C fusion Represents comprehensive characteristics, represents the attention weight of the kth fusion feature point, Represents the kth fused feature point.

[0040] Preferably, the performing multiple recognitions on the comprehensive features and post-processing the initial recognition results obtained to obtain the target recognition results includes:

[0041] Performing attention level recognition on the comprehensive features based on a long short-term memory network to obtain an initial student attention level;

[0042] Performing participation recognition on the comprehensive features based on a convolutional neural network to obtain initial student participation;

[0043] Based on the graph convolutional neural network, the interaction status of the comprehensive features is identified to obtain the initial student interaction status;

[0044] The initial student attention level, initial student participation and initial student interaction are denoised and outliers are removed respectively to obtain the target student attention level, target student participation and target student interaction.

[0045] Preferably, the quantitative evaluation of the target recognition result and the generation of a behavior analysis report include:

[0046] Calculating scores for the target student's attention level, target student participation, and target student interaction respectively, and standardizing the calculated scores to obtain a quantitative value for the student's attention level, a quantitative value for the student's participation, and a quantitative value for the student's interaction;

[0047] The quantified values ​​of the student's attention level, the quantified values ​​of the student's participation level and the quantified values ​​of the student's interaction are summarized to generate a behavior analysis report.

[0048] In a second aspect, an embodiment of the present invention provides an AI classroom behavior collaborative analysis system, including:

[0049] A data acquisition module is used to collect classroom environment data in real time, wherein the classroom environment data includes initial student video data, initial student audio data, initial student motion data and initial student expression data, and the data acquisition module is composed of multiple intelligent agents, each of which is equipped with a video data acquisition device, an audio data acquisition device and a motion data acquisition device;

[0050] A data preprocessing module, used to preprocess the classroom environment data to obtain target student video data, target student audio data, target student action data and target student expression data;

[0051] A feature extraction module is used to extract features from the target student video data, the target student audio data, the target student action data and the target student expression data, and obtain expression features, speech content features, action features and posture features respectively;

[0052] A comprehensive processing module, used for comprehensively processing the expression features, speech content features, action features and posture features to obtain comprehensive features, wherein the comprehensive processing includes single-modal attention mechanism processing, feature fusion and multi-modal attention mechanism processing;

[0053] A target recognition result determination module is used to perform multiple recognition on the comprehensive features and post-process the initial recognition results obtained to obtain target recognition results, wherein the multiple recognitions include attention level recognition, participation recognition and interaction status recognition, and the target recognition results include target student attention level, target student participation and target student interaction status;

[0054] The quantitative evaluation module is used to perform quantitative evaluation on the target recognition result and generate a behavior analysis report, wherein the behavior analysis report includes a quantitative value of the student's attention level, a quantitative value of the student's participation level, and a quantitative value of the student's interaction status.

[0055] Compared with the prior art, the AI ​​classroom behavior collaborative analysis method and system of the embodiment of the present invention has the following beneficial effects: it can collect video, audio, action and expression data in the classroom environment in real time, and realize real-time synchronization and time alignment of multi-source data to ensure the accuracy and consistency of the data; introduce a multimodal attention mechanism to perform weighted processing on the features of each modality, make full use of the complementarity of multi-source data, and improve the accuracy and effectiveness of feature fusion; realize comprehensive analysis of video data, audio data, action data and expression data, and improve the accuracy and comprehensiveness of behavior recognition and evaluation; perform multiple recognition on the extracted comprehensive features to obtain students' attention level, participation and interaction, and perform quantitative evaluation to ensure the reliability of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flowchart of an AI classroom behavior collaborative analysis method according to an embodiment of the present invention;

[0057] Figure 2 is a schematic diagram of a process of preprocessing classroom environment data according to an embodiment of the present invention;

[0058] Figure 3 It is a schematic diagram of the feature extraction process of an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of a flow chart of obtaining speech content features according to an embodiment of the present invention;

[0060] Figure 5 It is a schematic diagram of the comprehensive processing flow of an embodiment of the present invention;

[0061] Figure 6 is a schematic diagram of a process for obtaining a target recognition result according to an embodiment of the present invention;

[0062] Figure 7 This is another flowchart of an AI classroom behavior collaborative analysis method according to an embodiment of the present invention;

[0063] Figure 8It is a structural diagram of an AI classroom behavior collaborative analysis system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0065] In the description of the present invention, it should be understood that the terms "first" and "second" etc. are used in the present invention to distinguish different objects rather than to describe a specific order.

[0066] like Figure 1 As shown, an embodiment of the present invention provides an AI classroom behavior collaborative analysis method, comprising the steps of:

[0067] S1, real-time collection of classroom environment data;

[0068] The present invention configures multiple agents, each of which is equipped with a camera, a microphone and a motion sensor. The multiple agents collect classroom environment data in real time, and the classroom environment data includes initial student video data, initial student audio data, initial student motion data and initial student expression data.

[0069] Each agent collects the initial student video data in the classroom environment in real time through the camera. The following formula is used to represent the initial student video data:

[0070]

[0071] Among them, V i (t) represents the initial student video data collected by the i-th agent at time t, W and H represent the width and height of the video frame respectively, and C xy (t) represents the color information of the pixel point (x, y) at time t, L xy (t) represents the brightness information of the pixel point (x, y) at time t, and f(·) represents a complex function that combines color information and brightness information and is used to extract key video features.

[0072] Each agent collects the initial student audio data in the classroom environment in real time through a microphone. The following formula is used to represent the initial student audio data:

[0073]

[0074] Among them, A i (t) represents the initial student audio data collected by the i-th agent at time t, T represents the sampling duration, and F i (f, t) represents the spectrum of the audio signal at frequency f and time t, SNR i(t) represents the signal-to-noise ratio, and g(·) represents a complex function that combines the spectrum and the signal-to-noise ratio to extract key audio features.

[0075] Each agent collects the initial student motion data in the classroom environment in real time through motion sensors. The following formula is used to represent the initial student motion data:

[0076]

[0077] Among them, M i (t) represents the initial student action data collected by the i-th agent at time t, J represents the number of action sensors, and A ij (t) represents the acceleration data of the jth motion sensor at time t, V ij (t) represents the velocity data of the jth motion sensor at time t, and h(·) represents a complex function that combines acceleration information and velocity information to extract key motion features.

[0078] Each agent collects the initial student expression data in the classroom environment in real time through the camera. The following formula is used to represent the initial student expression data:

[0079]

[0080] Among them, E i (t) represents the initial student expression data collected by the i-th agent at time t, K represents the number of expression feature points, and F k (t) represents the coordinates of the kth facial feature point at time t, A k (t) represents the angle of the kth facial feature point at time t, S k (t) represents the scale change of the kth expression feature point at time t, and p(·) represents a complex function that combines coordinates, angles, and scale changes, which is used to extract key expression features.

[0081] S2, preprocessing the classroom environment data to obtain target student video data, target student audio data, target student action data and target student expression data;

[0082] Specifically, Figure 2 As shown, step S2 includes:

[0083] S201, time-synchronize the initial student video data, the initial student audio data, the initial student action data and the initial student expression data to obtain the intermediate student video data, the intermediate student audio data, the intermediate student action data and the intermediate student expression data;

[0084] The following is a detailed description of the time synchronization process:

[0085] 1) Using a data synchronization algorithm to perform preliminary time marking on the initial student video data, initial student audio data, initial student action data, and initial student expression data;

[0086] The marked time is the timestamp when the corresponding data is collected.

[0087] 2) Using a timestamp alignment algorithm to perform time alignment processing on the video data, audio data, action data and expression data with preliminary time marks, to obtain the time-aligned video data, audio data, action data and expression data;

[0088] 3) using an interpolation algorithm to perform interpolation processing on the time-aligned video data, audio data, motion data and expression data to obtain interpolated video data, audio data, motion data and expression data;

[0089] 4) Use a correction algorithm to perform time correction on the interpolated video data, audio data, motion data and expression data to obtain corrected video data, audio data, motion data and expression data, namely, intermediate student video data, intermediate student audio data, intermediate student motion data and intermediate student expression data.

[0090] S202, performing denoising and filtering processing on the middle student video data, the middle student audio data, the middle student action data and the middle student expression data to obtain the target student video data, the target student audio data, the target student action data and the target student expression data respectively.

[0091] The following is a detailed description of the denoising filtering process:

[0092] 1) De-noising the middle student video data, using a Gaussian filter to smooth the middle student video data to obtain the denoised video data, i.e., the target student video data;

[0093] 2) De-noising the middle student audio data, using a bandpass filter to smooth the middle student audio data to obtain de-noised audio data, i.e., target student audio data;

[0094] 3) De-noising the intermediate student action data, and using a Kalman filter to smooth the intermediate student action data to obtain the denoised action data, i.e., the target student action data;

[0095] 4) De-noising the middle student expression data, and smoothing the middle student expression data using a median filter to obtain the denoised expression data, i.e., the target student expression data.

[0096] S3, extracting features from the target student video data, the target student audio data, the target student action data and the target student expression data, and obtaining expression features, speech content features, action features and posture features respectively;

[0097] Specifically, Figure 3 As shown, step S3 includes:

[0098] S301, performing a first feature extraction on the target student video data and the target student expression data to obtain expression features and posture features;

[0099] The target student video data and the target student expression data are input into the improved hybrid computer vision model for first feature extraction to obtain expression features and posture features. The improved hybrid computer vision model combines a convolutional neural network based on an attention mechanism and a graph convolutional neural network, and the first feature extraction includes attention mechanism processing, graph convolution operation and feature fusion.

[0100] The following is a detailed description of the first feature extraction process:

[0101] 1) The convolutional neural network based on the attention mechanism uses the attention mechanism to perform weighted processing on the target student video data and extract the initial expression features;

[0102] In the improved hybrid computer vision model, a convolutional neural network based on the attention mechanism is used to focus on the important areas in the target student’s video data. The following formula is used to characterize the initial expression features:

[0103]

[0104] in, represents the initial expression feature, α k represents the attention weight, f k represents the convolution operation, represents the target student video data, e k represents the score function, w, W1, W2, b represent the model parameters, h k-1 Represents the hidden state of the previous convolutional neural network.

[0105] 2) The graph convolutional neural network performs graph convolution operations on the target student’s expression data to extract initial posture features;

[0106] In the improved hybrid computer vision model, the graph convolutional neural network is used to process the spatial relationship of the target student's expression data. The graph convolutional neural network processes the relationship between facial key points through graph convolution operations and extracts the initial posture features. The following formula is used to characterize the initial posture features:

[0107]

[0108] in, represents the initial posture feature, N(i) represents the neighbor node set of the i-th node, c ij represents the normalization coefficient between node i and node j, W represents the weight matrix, represents the target student's expression data, b represents the bias, and σ represents the activation function.

[0109] 3) Fusing the initial expression features and initial posture features to obtain a comprehensive feature vector;

[0110] The following formula is used to characterize the comprehensive feature vector:

[0111]

[0112] in, represents the comprehensive feature vector, and ψ represent the transformation functions of the initial expression features and initial posture features respectively. Represents a feature concatenation operation.

[0113] 4) Optimize the comprehensive feature vector to obtain expression features and posture features.

[0114] Specifically, the optimization process includes removing noise and outliers. The following formula is used to characterize the expression features and posture features:

[0115]

[0116] in, represents expression features and posture features, θ represents the optimization function, μ and σ represent the mean and standard deviation of the comprehensive feature vector respectively.

[0117] S302, extracting the second feature of the target student's audio data to obtain a speech content feature;

[0118] The target student audio data is input into the improved self-supervised learning speech recognition model for second feature extraction to obtain speech content features. The improved self-supervised learning speech recognition model includes a Transformer network, an attention mechanism, and a self-supervised learning module, and the second feature extraction includes global feature extraction, key feature extraction, and self-supervised learning.

[0119] Specifically, Figure 4 As shown, step S302 includes:

[0120] S302-A, extracting global features of the target student's audio data based on the Transformer network to obtain global features of the speech content;

[0121] In the improved self-supervised learning speech recognition model, the Transformer network is used to extract global features of the target student audio data.

[0122] S302-B, extracting key features from the global features of the speech content based on the attention mechanism to obtain key features of the speech content;

[0123] The global features of the speech content are input into the attention mechanism to further extract the key features of the speech content.

[0124] S302-C, performing self-supervisory learning on key features of the speech content to obtain speech content features.

[0125] The extracted key features of the speech content are input into the self-supervised learning module to perform speech content recognition and obtain speech content features. The following formula is used to characterize the speech content features:

[0126]

[0127] in, Indicates the characteristics of speech content. represents the key features of speech content, and SSL represents the self-supervised learning module.

[0128] The self-supervised learning module calculation is as follows:

[0129]

[0130] Among them, L pretext and L finetune denote the total loss function of the pre-training task and the fine-tuning task respectively, and Represent the loss functions of the pre-training task and the fine-tuning task, respectively, θ and Represent the models of the pre-training task and the fine-tuning task, y i and z i denote the objectives of the pre-training task and the fine-tuning task, respectively.

[0131] After step S302-C, the speech content features are post-processed, including removing noise and correcting errors, to obtain the final optimized speech content features.

[0132] S303, extracting the third feature of the target student's motion data to obtain motion features. The target student's motion data is input into the improved motion recognition model to extract the third feature to obtain motion features. The improved motion recognition model combines the spatiotemporal convolutional network and the long short-term memory network, and the third feature extraction includes spatiotemporal feature extraction, time series feature extraction and motion classification.

[0133] The following is a detailed description of the process of extracting the third feature:

[0134] 1) Input the target student's action data into the spatiotemporal convolutional network to extract spatiotemporal features and obtain spatiotemporal features;

[0135] In the improved action recognition model, the spatiotemporal convolutional network is used to extract the spatiotemporal features of the target student's action data.

[0136] 2) Input the spatiotemporal features into the long short-term memory network to extract the time series features and obtain the time series features of the target student's action data;

[0137] 3) Input the time series features into the fully connected layer for action classification to obtain action features.

[0138] After 3), the action features are post-processed, including removing noise and correcting errors, to obtain the final optimized action features.

[0139] S4, comprehensively processing the expression features, speech content features, action features and posture features to obtain comprehensive features;

[0140] Based on the collaborative mechanism of multiple agents, the extracted expression features, speech content features, action features and posture features are comprehensively processed. The comprehensive processing includes single-modal attention mechanism processing, feature fusion and multi-modal attention mechanism processing.

[0141] Specifically, Figure 5 As shown, step S4 includes:

[0142] S401, performing unimodal attention mechanism processing on expression features, speech content features, action features and posture features respectively, to obtain weighted expression features, weighted speech content features, weighted action features and weighted posture features;

[0143] The corresponding unimodal attention mechanism is applied to the features of each modality, that is, the expression attention mechanism is applied to the expression features, the auditory attention mechanism is applied to the speech content features, the action attention mechanism is applied to the action features, and the posture attention mechanism is applied to the posture features.

[0144] Specifically, the first preset formula is used to characterize the unimodal attention mechanism processing, and the first preset formula is as follows:

[0145]

[0146] Among them, F att represents weighted expression feature or weighted speech content feature or weighted action feature or weighted posture feature, K represents the total number of expression feature points or the total number of speech content feature points or the total number of action feature points or the total number of posture feature points, α krepresents the attention weight of the kth expression feature point, the attention weight of the kth speech content feature point, the attention weight of the kth action feature point, or the attention weight of the kth posture feature point, f k represents the kth expression feature point or the kth speech content feature point or the kth action feature point or the kth posture feature point, and F represents the expression feature or the speech content feature or the action feature or the posture feature.

[0147] Further, there are:

[0148]

[0149] e k =w T tanh(WF+b)

[0150] Among them, e k represents the score function of the k-th expression feature point, the score function of the k-th speech content feature point, the score function of the k-th action feature point, or the score function of the k-th posture feature point, w, W, b represent the model parameters of the expression attention mechanism, the model parameters of the auditory attention mechanism, the model parameters of the action attention mechanism, or the model parameters of the posture attention mechanism.

[0151] S402, performing feature fusion on the weighted expression feature, the weighted speech content feature, the weighted action feature and the weighted posture feature to obtain a fusion feature;

[0152] The features processed by the unimodal attention mechanism are fused to obtain fused features.

[0153] Specifically, the second preset formula is used to represent feature fusion, and the second preset formula is as follows:

[0154]

[0155] Among them, F fusion represents the fusion feature, represents the feature concatenation operation, represents the weighted posture feature, represents the weighted speech content features, represents the weighted action feature, Represents weighted expression features.

[0156] S403, performing multimodal attention mechanism processing on the fused features to obtain comprehensive features.

[0157] Specifically, the third preset formula is used to characterize the multimodal attention mechanism processing, and the third preset formula is as follows:

[0158]

[0159] Among them, C fusion Represents comprehensive characteristics, represents the attention weight of the kth fusion feature point, Represents the kth fused feature point.

[0160] Further, there are:

[0161]

[0162] in, represents the score function of the kth fusion feature point, w fusion , W fusion 、b fusion Represents the model parameters of the multimodal attention mechanism.

[0163] S5, performing multiple recognition on the comprehensive features, and post-processing the initial recognition results obtained to obtain the target recognition results;

[0164] The comprehensive features are input into the downstream task model for multiple recognition, and then the target recognition results are obtained. Among them, multiple recognition includes attention level recognition, participation recognition and interaction recognition. The target recognition results include the target student's attention level, target student participation and target student interaction.

[0165] Specifically, if Figure 6 As shown, step S5 includes:

[0166] S501, identifying the attention level of the comprehensive features based on the long short-term memory network to obtain the initial student attention level;

[0167] Based on the long short-term memory network, it is used to capture the long-term dependencies of students’ behavioral characteristics over time and analyze students’ attention levels in different time periods, that is, the initial students’ attention levels.

[0168] S502, identifying the degree of participation of the comprehensive features based on a convolutional neural network to obtain an initial degree of student participation;

[0169] Based on convolutional neural networks, it is used to extract students' behavioral characteristics in the classroom, including the spatial characteristics of voice, action and expression, and identify students' participation, namely the initial student participation.

[0170] S503, identifying the interaction status of the comprehensive features based on the graph convolutional neural network to obtain the initial student interaction status;

[0171] Based on graph convolutional neural network, it is used to capture the interactive behavior characteristics of students in the classroom and analyze the interactive relationship between students and teachers and other students, that is, the initial student interaction situation.

[0172] S504, denoising and removing outliers are performed on the initial student attention level, the initial student participation and the initial student interaction to obtain the target student attention level, the target student participation and the target student interaction.

[0173] It is understood that post-processing includes denoising and outlier removal.

[0174] S6. Quantitatively evaluate the target recognition results and generate a behavior analysis report.

[0175] The core of the quantitative assessment is to convert the target student's attention level, target student participation and target student interaction identified in step S5 into corresponding quantitative values, generate a behavior analysis report, and realize the quantitative analysis of these behavior characteristics. The behavior analysis report includes the quantitative values ​​of the student's attention level, the quantitative values ​​of the student's participation and the quantitative values ​​of the student's interaction.

[0176] The following is a detailed description of the quantitative evaluation process:

[0177] 1) Calculate the scores of the target students' attention level, target students' participation and target students' interaction respectively, and standardize the calculated scores to obtain the quantitative values ​​of the students' attention level, the quantitative values ​​of the students' participation and the quantitative values ​​of the students' interaction;

[0178] This embodiment maps the target student's attention level, target student participation, and target student interaction to a scoring range of 0 to 100. Specifically, the score of the target student's attention level is generated based on the weight distribution and contribution of the comprehensive features in the long short-term memory network, the score of the target student's participation is generated based on the weight distribution and contribution of the comprehensive features in the convolutional neural network, and the score of the target student's interaction is generated based on the weight distribution and contribution of the comprehensive features in the graph convolutional neural network. Furthermore, the generated scores are standardized to ensure their consistency and comparability between different students or different classrooms. This embodiment is achieved through a normalization algorithm to ensure that the score range is stable and there is no data bias. Standardized scores are easier to interpret and apply, and teachers can judge students' overall classroom performance and individual differences based on this.

[0179] 2) Summarize the quantitative values ​​of students’ attention levels, students’ participation levels, and students’ interactions to generate a behavior analysis report.

[0180] These quantitative values ​​are summarized in a behavior analysis report. The report presents the scores of students' attention level, participation and interaction in the form of charts and numerical values, showing their trend changes over a period of time. Important time nodes are marked in the report, such as the moment when attention suddenly decreases or participation increases significantly, so that teachers can make timely teaching adjustments.

[0181] It should be noted that step S6 aims to convert complex behavior recognition results into quantitative data that is easy to understand and apply, help teachers obtain real-time performance information of students in class, and provide data support to optimize teaching methods and improve classroom interaction effects.

[0182] Furthermore, if Figure 7 As shown, after step S6, the following steps are also included:

[0183] S7. Provide real-time feedback and teaching suggestions based on behavior analysis reports.

[0184] Provide teachers with real-time feedback and teaching suggestions based on behavioral analysis reports, including prompting students who need more attention and suggesting adjustments to teaching methods.

[0185] The embodiment of the present invention provides an AI classroom behavior collaborative analysis method, which can collect video, audio, action and expression data in a classroom environment in real time, and realize real-time synchronization and time alignment of multi-source data to ensure the accuracy and consistency of the data; introduce a multimodal attention mechanism, perform weighted processing on the features of each modality, make full use of the complementarity of multi-source data, and improve the accuracy and effectiveness of feature fusion; realize comprehensive analysis of video data, audio data, action data and expression data, and improve the accuracy and comprehensiveness of behavior recognition and evaluation; perform multiple recognition on the extracted comprehensive features to obtain students' attention level, participation and interaction, and perform quantitative evaluation to ensure the reliability of the evaluation results.

[0186] In order to verify the effectiveness of an AI classroom behavior collaborative analysis method according to an embodiment of the present invention, the above-mentioned AI classroom behavior collaborative analysis method is applied in a specific embodiment.

[0187] In a certain university, teachers hope to improve the quality of classroom teaching and students' learning experience through technical means. The school selected the AI ​​classroom behavior collaborative analysis method provided by the present invention to monitor and analyze students' classroom behavior in real time. Through a variety of devices such as cameras, microphones and motion sensors, students' expressions, postures, voices and motion behavior data are collected in real time. The experiment was conducted in a class with a total of 30 students. The experimental time was from March 2024 to June 2024, for a period of 3 months.

[0188] Before the class started, multiple agents were deployed in various corners of the classroom. Each agent was equipped with a camera, microphone, and motion sensor to ensure full coverage of the entire classroom and to collect various behavioral data of students in real time. The data collected by each agent was time-aligned to ensure data consistency.

[0189] In class, teachers use the system to analyze students' behaviors in real time. The system first performs denoising and filtering on the collected video data, audio data, action data, and expression data. Subsequently, the preprocessed data is input into the improved hybrid computer vision model, the improved self-supervised learning speech recognition model, and the improved action recognition model to extract students' expression features, posture features, speech content features, and action features, respectively. Through the multimodal attention mechanism, these features are fused to form comprehensive features.

[0190] During classroom teaching, the system can analyze students' attendance rate, front-row seating rate, and classroom attention, and also provide micro-individual attention analysis to identify some students who have been out of classroom teaching for a long time, showing inattention and low participation. For example, through comprehensive analysis of facial expression and posture features, the system found that the three students sitting in the back row frequently lowered their heads, had dull expressions, and had low attention levels; through comprehensive analysis of voice content features and movement features, the system found that a student in the front row participated less in discussions and had poor interaction. The system feeds back these analysis results to teachers in real time, and teachers adjust their teaching strategies accordingly, increase classroom interaction, and pay special attention to the performance of these students.

[0191] During the experiment, the system recorded the students' behavior data and generated a detailed behavior analysis report. The comparison data between the method of the present invention and the traditional method are shown in Table 1.

[0192] Table 1 Comparison data between traditional method and method of the present invention

[0193]

[0194]

[0195] During the experiment, the system monitored and analyzed the student behavior in each class, and collected data from 60 classes in total. The following are some specific data:

[0196] Week 1: The system identified that the front row seating rate was low and the attendance rate was not high. After the teacher adjusted the teaching strategy, the front row seating rate and attendance rate identified in the second week improved significantly.

[0197] Week 4: The system found that four students had low participation. By increasing the number of interactive sessions in class, the participation of these four students increased significantly after the fifth week, and the interaction was good.

[0198] Week 8: The system identified two students with poor interaction. After targeted guidance from the teacher, the interaction between the two students improved significantly in week 9.

[0199] At the end of the experiment, teachers and students expressed high satisfaction with the feedback from the system. Teachers believe that the system not only improves the efficiency of classroom management, but also can provide targeted teaching suggestions in real time, greatly improving the quality of teaching. Students also said that in this intelligent classroom environment, they can better concentrate, participate in classroom interaction, and significantly improve learning effects. Through the method of the present invention, the school significantly improved the efficiency of classroom management and teaching optimization in a short period of time, and greatly improved the students' learning experience. Compared with traditional methods, the method of the present invention has obvious advantages in data collection, processing and analysis, and can identify and evaluate students' behavior in real time and accurately, providing an efficient and intelligent solution for modern education.

[0200] Based on the above AI classroom behavior collaborative analysis method, Figure 8 As shown, an embodiment of the present invention provides an AI classroom behavior collaborative analysis system, including:

[0201] Data collection module 1, used for collecting classroom environment data in real time, wherein the classroom environment data includes initial student video data, initial student audio data, initial student action data and initial student expression data;

[0202] Specifically, the data acquisition module 1 is composed of multiple agents, each of which is equipped with a video data acquisition device, an audio data acquisition device and a motion data acquisition device. In this embodiment, the video data acquisition device is a camera, the audio data acquisition device is a microphone, and the motion data acquisition device is a motion sensor.

[0203] Data preprocessing module 2, used for preprocessing classroom environment data to obtain target student video data, target student audio data, target student action data and target student expression data;

[0204] Feature extraction module 3, used for extracting features from target student video data, target student audio data, target student action data and target student expression data, and obtaining expression features, speech content features, action features and posture features respectively;

[0205] Comprehensive processing module 4, used for comprehensive processing of expression features, speech content features, action features and posture features to obtain comprehensive features, wherein the comprehensive processing includes single-modal attention mechanism processing, feature fusion and multi-modal attention mechanism processing;

[0206] The target recognition result determination module 5 is used to perform multiple recognition on the comprehensive features and post-process the initial recognition results obtained to obtain the target recognition results, wherein the multiple recognition includes attention level recognition, participation recognition and interaction status recognition, and the target recognition results include the target student's attention level, target student participation and target student interaction status;

[0207] The quantitative evaluation module 6 is used to perform quantitative evaluation on the target recognition result and generate a behavior analysis report, wherein the behavior analysis report includes the quantitative value of the student's attention level, the quantitative value of the student's participation level and the quantitative value of the student's interaction status.

[0208] It should be noted that the various modules except the data acquisition module in the above-mentioned AI classroom behavior collaborative analysis system can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned various modules except the data acquisition module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the various modules except the data acquisition module. For the specific definition of an AI classroom behavior collaborative analysis system, please refer to the definition of an AI classroom behavior collaborative analysis method above. The two have the same functions and effects, which will not be repeated here.

[0209] In summary, the embodiment of the present invention provides an AI classroom behavior collaborative analysis method and system, which can collect video, audio, action and expression data in the classroom environment in real time, and realize real-time synchronization and time alignment of multi-source data to ensure the accuracy and consistency of the data; introduce a multimodal attention mechanism, perform weighted processing on the features of each modality, make full use of the complementarity of multi-source data, and improve the accuracy and effectiveness of feature fusion; realize comprehensive analysis of video data, audio data, action data and expression data, and improve the accuracy and comprehensiveness of behavior recognition and evaluation; perform multiple recognition on the extracted comprehensive features to obtain students' attention level, participation and interaction, and perform quantitative evaluation to ensure the reliability of the evaluation results.

[0210] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A collaborative analysis method for AI classroom behavior, characterized in that: include: Collecting classroom environment data in real time, wherein the classroom environment data includes initial student video data, initial student audio data, initial student action data and initial student expression data; Preprocessing the classroom environment data to obtain target student video data, target student audio data, target student action data, and target student expression data; Extracting features from the target student video data, the target student audio data, the target student action data, and the target student expression data to obtain expression features, speech content features, action features, and posture features, respectively; Comprehensively processing the expression features, speech content features, action features and posture features to obtain comprehensive features, wherein the comprehensive processing includes single-modal attention mechanism processing, feature fusion and multi-modal attention mechanism processing; Perform multiple recognition on the comprehensive features, and post-process the initial recognition results obtained to obtain target recognition results, wherein the multiple recognitions include attention level recognition, participation recognition, and interaction status recognition, and the target recognition results include the target student's attention level, the target student's participation, and the target student's interaction status; The target recognition result is quantitatively evaluated to generate a behavior analysis report, wherein the behavior analysis report includes a quantitative value of the student's attention level, a quantitative value of the student's participation level, and a quantitative value of the student's interaction status.

2. The AI ​​classroom behavior collaborative analysis method according to claim 1 is characterized in that: After the target recognition result is quantitatively evaluated and a behavior analysis report is generated, the method further includes: Real-time feedback and teaching suggestions are provided based on the behavior analysis report.

3. The AI ​​classroom behavior collaborative analysis method according to claim 1 is characterized in that: The preprocessing of the classroom environment data to obtain target student video data, target student audio data, target student action data and target student expression data respectively includes: Performing time synchronization on the initial student video data, the initial student audio data, the initial student action data and the initial student expression data to obtain intermediate student video data, intermediate student audio data, intermediate student action data and intermediate student expression data; The intermediate student video data, intermediate student audio data, intermediate student action data and intermediate student expression data are subjected to denoising and filtering processing to obtain target student video data, target student audio data, target student action data and target student expression data respectively.

4. The AI ​​classroom behavior collaborative analysis method according to claim 1 is characterized in that: The feature extraction of the target student video data, the target student audio data, the target student action data and the target student expression data is performed to obtain expression features, speech content features, action features and posture features, respectively, including: Performing a first feature extraction on the target student video data and the target student expression data to obtain expression features and posture features, wherein the first feature extraction includes attention mechanism processing, graph convolution operation and feature fusion; Performing a second feature extraction on the target student audio data to obtain speech content features, wherein the second feature extraction includes global feature extraction, key feature extraction and self-supervised learning; Performing a third feature extraction on the target student's motion data to obtain motion features, wherein the third feature extraction includes spatiotemporal feature extraction, time series feature extraction and motion classification.

5. The AI ​​classroom behavior collaborative analysis method according to claim 4 is characterized in that: The second feature extraction is performed on the target student audio data to obtain the speech content feature, including: Performing global feature extraction on the target student audio data based on the Transformer network to obtain global features of the speech content; Extract key features of the global features of the speech content based on an attention mechanism to obtain key features of the speech content; Self-supervised learning is performed on the key features of the speech content to obtain speech content features.

6. The AI ​​classroom behavior collaborative analysis method according to claim 1 is characterized in that: The expression features, voice content features, action features and posture features are comprehensively processed to obtain comprehensive features, including: The expression features, speech content features, action features and posture features are processed by a unimodal attention mechanism to obtain weighted expression features, weighted speech content features, weighted action features and weighted posture features; Performing feature fusion on the weighted expression feature, weighted speech content feature, weighted action feature and weighted posture feature to obtain a fusion feature; The fused features are processed by a multimodal attention mechanism to obtain comprehensive features.

7. The AI ​​classroom behavior collaborative analysis method according to claim 6 is characterized in that: The unimodal attention mechanism processing is characterized by a first preset formula, and the first preset formula is as follows: Among them, F att represents weighted expression feature or weighted speech content feature or weighted action feature or weighted posture feature, K represents the total number of expression feature points or the total number of speech content feature points or the total number of action feature points or the total number of posture feature points, α k represents the attention weight of the kth expression feature point, the attention weight of the kth speech content feature point, the attention weight of the kth action feature point, or the attention weight of the kth posture feature point, f k represents the kth expression feature point or the kth speech content feature point or the kth action feature point or the kth posture feature point, and F represents the expression feature or the speech content feature or the action feature or the posture feature; A second preset formula is used to characterize the feature fusion, and the second preset formula is as follows: Among them, F fusion represents the fusion feature, represents the feature concatenation operation, represents the weighted posture feature, represents the weighted speech content features, represents the weighted action feature, represents weighted expression features; The multimodal attention mechanism processing is characterized by a third preset formula, and the third preset formula is as follows: Among them, C fusion Represents comprehensive characteristics, represents the attention weight of the kth fusion feature point, Represents the kth fused feature point.

8. The AI ​​classroom behavior collaborative analysis method according to claim 1 is characterized in that: The multiple recognition of the comprehensive features and post-processing of the initial recognition results obtained to obtain the target recognition results include: Performing attention level recognition on the comprehensive features based on a long short-term memory network to obtain an initial student attention level; Performing participation recognition on the comprehensive features based on a convolutional neural network to obtain initial student participation; Based on the graph convolutional neural network, the interaction status of the comprehensive features is identified to obtain the initial student interaction status; The initial student attention level, initial student participation and initial student interaction are denoised and outliers are removed respectively to obtain the target student attention level, target student participation and target student interaction.

9. The AI ​​classroom behavior collaborative analysis method according to claim 1, characterized in that: The quantitative evaluation of the target recognition result and the generation of a behavior analysis report include: Calculating scores for the target student's attention level, target student participation, and target student interaction respectively, and standardizing the calculated scores to obtain a quantitative value for the student's attention level, a quantitative value for the student's participation, and a quantitative value for the student's interaction; The quantified values ​​of the student's attention level, the quantified values ​​of the student's participation level and the quantified values ​​of the student's interaction are summarized to generate a behavior analysis report.

10. An AI classroom behavior collaborative analysis system, characterized in that: include: A data acquisition module is used to collect classroom environment data in real time, wherein the classroom environment data includes initial student video data, initial student audio data, initial student motion data and initial student expression data, and the data acquisition module is composed of multiple intelligent agents, each of which is equipped with a video data acquisition device, an audio data acquisition device and a motion data acquisition device; A data preprocessing module, used to preprocess the classroom environment data to obtain target student video data, target student audio data, target student action data and target student expression data; A feature extraction module is used to extract features from the target student video data, the target student audio data, the target student action data and the target student expression data, and obtain expression features, speech content features, action features and posture features respectively; A comprehensive processing module, used for comprehensively processing the expression features, speech content features, action features and posture features to obtain comprehensive features, wherein the comprehensive processing includes single-modal attention mechanism processing, feature fusion and multi-modal attention mechanism processing; A target recognition result determination module is used to perform multiple recognition on the comprehensive features and post-process the initial recognition results obtained to obtain target recognition results, wherein the multiple recognitions include attention level recognition, participation recognition and interaction status recognition, and the target recognition results include target student attention level, target student participation and target student interaction status; The quantitative evaluation module is used to perform quantitative evaluation on the target recognition result and generate a behavior analysis report, wherein the behavior analysis report includes a quantitative value of the student's attention level, a quantitative value of the student's participation level, and a quantitative value of the student's interaction status.

Images