Classroom interaction behavior recognition method and system based on causal reasoning

Abstract

The application provides a classroom interactive behavior recognition method and system based on causal reasoning, and the method comprises the following steps: obtaining an interactive video input by a user, wherein the interactive video comprises individual behaviors in a classroom scene, and the individual behaviors comprise teacher behaviors and student behaviors; inputting the interactive video into a classroom interactive behavior recognition model to obtain recognition results of an interactive sequence and recognition results of the individual behaviors output by the classroom interactive behavior recognition model; the interactive sequence is used to represent an alternating mode or an interactive mode between the teacher behaviors and the student behaviors; the classroom interactive behavior recognition model is obtained based on interactive video samples, real class labels of the interactive sequences corresponding to the interactive video samples, and real class labels of the individual behaviors corresponding to the interactive video samples; and the application can capture various behaviors occurring continuously in one classroom interactive event, and the dynamic nature of the interaction is exhibited.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a classroom interaction behavior recognition method and system based on causal reasoning. BACKGROUND

[0002] Artificial intelligence technology has made significant progress in behavior recognition, automatic translation, autonomous driving and many other fields, greatly facilitating the rapid and accurate extraction of information from a large amount of video, image and text data. In the field of education, classroom interaction behavior is an important indicator for evaluating teaching quality, providing a basis for accurately grasping student needs and adjusting teaching strategies.

[0003] However, there are still some problems in the recognition of classroom interaction behavior by artificial intelligence. Existing classroom behavior datasets usually only focus on the independent behavior of a specific participant (teacher / student), such as raising a hand or sleeping, ignoring the dynamic nature of interaction between participants, which negatively affects the evaluation of interaction quality.

[0004] Therefore, the classroom interaction behavior recognition method in the related art has the technical problem of ignoring the dynamic nature of interaction between participants in the classroom. SUMMARY

[0005] The present application provides a classroom interaction behavior recognition method and system based on causal reasoning, which solves the problem of ignoring the dynamic nature of interaction between participants in the classroom in the prior art, and captures various behaviors occurring continuously in a classroom interaction event, showing the dynamic nature of interaction.

[0006] The present application provides a classroom interaction behavior recognition method based on causal reasoning, comprising the following steps. An interaction video input by a user is obtained, wherein the interaction video includes individual behavior in a classroom scene, and the individual behavior includes teacher behavior and student behavior. The interaction video is input into a classroom interaction behavior recognition model to obtain an identification result of an interaction sequence and an identification result of individual behavior output by the classroom interaction behavior recognition model. The interaction sequence is used to represent the alternating mode or interaction mode between the teacher behavior and the student behavior. The classroom interaction behavior recognition model is trained based on an interaction video sample, a real class label of the interaction sequence corresponding to the interaction video sample, and a real class label of individual behavior corresponding to the interaction video sample.

[0007] According to the classroom interaction behavior recognition method based on causal reasoning provided by the application, the classroom interaction behavior recognition model comprises an individual behavior capturing module, a causal relationship constructing module, a first full connection layer, a maximum pooling layer and a second full connection layer, the interaction video is input into the classroom interaction behavior recognition model, and the recognition result of the interaction sequence and the recognition result of the individual behavior output by the classroom interaction behavior recognition model are obtained, which comprises: performing feature recognition on the interaction video through the individual behavior capturing module to obtain teacher behavior features and student behavior features; constructing the space-time correlation features between the teacher behavior features and the student behavior features through the causal relationship constructing module; performing feature fusion based on the teacher behavior features, the student behavior features and the space-time correlation features to obtain fusion features; performing causal intervention processing on the fusion features to obtain causal features, wherein the causal intervention processing is used to eliminate the false correlation of the fusion features; classifying the causal features through the first full connection layer to obtain the recognition result of the individual behavior; and classifying the causal features through the maximum pooling layer and the second full connection layer to obtain the recognition result of the interaction sequence.

[0008] According to the classroom interaction behavior recognition method based on causal reasoning provided by the application, the causal intervention processing on the fusion features to obtain causal features comprises: obtaining the average space context information between each behavior class pair in a single frame of the interaction video based on convolution of different scales; constructing a space matrix based on the average space context information, wherein the dimension of the space matrix is the product of the total number of behavior class pairs and the feature dimension of the space context of each behavior class pair; obtaining the average time context information between each behavior class pair in the interaction video based on convolution of different time steps; constructing a time matrix based on the average time context information, wherein the dimension of the time matrix is the product of the total number of behavior class pairs and the feature dimension of the time context of each behavior class pair; determining the space confusion dictionary corresponding to the space matrix and the time confusion dictionary corresponding to the time matrix; performing inner product and matrix multiplication on the space-time correlation features in the fusion features and the space confusion dictionary and the time confusion dictionary to obtain an expected value; and performing splicing and multi-scale gated feedforward processing on the space-time correlation features in the fusion features and the expected value to obtain causal features.

[0009] According to the classroom interaction behavior recognition method based on causal reasoning provided by the application, before the interaction video is input into the classroom interaction behavior recognition model to obtain the recognition result of the interaction sequence and the recognition result of individual behavior output by the classroom interaction behavior recognition model, the method further comprises: acquiring an interaction video sample, wherein the interaction video sample carries a real class label of the interaction sequence and a real class label of individual behavior; inputting the interaction video sample into a preset classroom interaction behavior recognition model to obtain a predicted class label of the interaction sequence and a predicted class label of individual behavior output by the preset classroom interaction behavior recognition model; updating the preset classroom interaction behavior recognition model based on a target loss function according to the real class label of the interaction sequence and the predicted class label of the interaction sequence, and the real class label of individual behavior and the predicted class label of individual behavior, to obtain a trained classroom interaction behavior recognition model.

[0010] According to the classroom interaction behavior recognition method based on causal reasoning provided by the application, the target loss function refers to the following formula:

[0011] ;

[0012] Wherein, The target loss function is represented by L, The first cross-entropy loss is represented by Lce, The real class label of the interaction sequence is represented by y, The predicted class label of the interaction sequence is represented by y, The first hyperparameter is represented by λ, The prediction accuracy for the interaction sequence is represented by acc, The second cross-entropy loss is represented by Lce, The real class label of individual behavior is represented by y, The predicted class label of individual behavior is represented by y, The second hyperparameter is represented by λ, The prediction accuracy for the individual behavior is represented by acc.

[0013] According to the classroom interaction behavior recognition method based on causal reasoning provided by the application, the interaction video sample comprises a plurality of individual behaviors and a plurality of interaction sequences; wherein the plurality of individual behaviors comprise: acceptance, rejection, questioning, discussion, active conversation, response, note taking or practice, teaching and guidance; the interaction behaviors of the plurality of interaction sequences comprise: teacher teaching and student listening, teacher instruction and student execution, question and answer, and group discussion.

[0014] The application further provides a classroom interactive behavior recognition system based on causal reasoning, comprising the following modules: an acquisition module, configured to acquire an interactive video input by a user, wherein the interactive video comprises individual behaviors in a classroom scene, and the individual behaviors comprise teacher behaviors and student behaviors; an identification module, configured to input the interactive video into a classroom interactive behavior recognition model to obtain an identification result of an interactive sequence output by the classroom interactive behavior recognition model and an identification result of the individual behaviors; the interactive sequence is used to represent an alternating mode or an interactive mode between the teacher behaviors and the student behaviors; and the classroom interactive behavior recognition model is trained based on interactive video samples, real class labels of the interactive sequences corresponding to the interactive video samples, and real class labels of the individual behaviors corresponding to the interactive video samples.

[0015] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the classroom interactive behavior recognition method based on causal reasoning according to any one of the above when executing the program.

[0016] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the classroom interactive behavior recognition method based on causal reasoning according to any one of the above.

[0017] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the classroom interactive behavior recognition method based on causal reasoning according to any one of the above.

[0018] The classroom interactive behavior recognition method and system based on causal reasoning provided by the application acquire an interactive video input by a user, wherein the interactive video comprises student behaviors and teacher behaviors, the interactive video is the interaction of participants in a classroom scene, the data is ensured to be targeted and relevant, and irrelevant information is avoided; the interactive video is input into a classroom interactive behavior recognition model, the classroom interactive behavior recognition model can accurately identify individual behaviors (including teacher behaviors and student behaviors) in a classroom scene and an interactive sequence formed by the behaviors, thereby the model not only identifies individual behaviors but also identifies interactive behaviors, the correlation and causal relationship between different behaviors can be analyzed by identifying the interactive sequence, which helps to deeply understand the classroom interactive mode; in addition, the classroom interactive behavior model can learn accurate features of classroom interactive behaviors by training a large number of interactive video samples and real class labels, and the recognition accuracy is improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced one by one as follows. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0020] Figure 1 is a model reasoning space-time confusion schematic diagram of the related technology provided by the present application.

[0021] Figure 2 is a flow schematic diagram of the classroom interaction behavior recognition method based on causal reasoning provided by the present application.

[0022] Figure 3 is a structural block diagram of the classroom interaction behavior recognition model based on causal reasoning provided by the present application.

[0023] Figure 4 is a framework schematic diagram of constructing a participant relationship graph provided by the present application.

[0024] Figure 5 is a schematic diagram of causal intervention provided by the present application.

[0025] Figure 6 is a structural schematic diagram of the multi-scale gating feedforward module provided by the present application.

[0026] Figure 7 is a flow schematic diagram of the causal intervention module provided by the present application.

[0027] Figure 8 is a schematic diagram of the actual scene from the perspective of students (a) and teachers (b) provided by the present application.

[0028] Figure 9 is a qualitative analysis schematic diagram on a visual interaction behavior dataset provided by the present application.

[0029] Figure 10 is a structural schematic diagram of the classroom interaction behavior recognition system based on causal reasoning provided by the present application.

[0030] Figure 11 is an entity structure schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application in conjunction with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection scope of the present application.

[0032] Classroom behavior recognition is often disturbed by visual disturbances such as light changes, occlusions, and simultaneous or sequential objects. This makes it challenging to achieve effective behavior representation. Although efforts have been made to introduce causal reasoning to address this issue, they ignore the unique semantic information embedded in the temporal and spatial confounders. In addition, existing classroom behavior datasets mainly focus on isolated actions, ignoring the dynamic of interactions. To overcome these limitations, the present application proposes a classroom interaction behavior recognition method based on causal reasoning. The spatial and temporal associations between behaviors are considered as confounders, and weights are assigned according to the probability of behavior occurrence. By applying multi-scale gated feedforward in causal intervention, the effects of temporal and spatial confounders can be eliminated. The sample dataset annotates interaction sequences that describe the dynamic process of behaviors in interaction events, enabling robust behavior analysis. Experimental results show that the classroom interaction behavior recognition method based on causal reasoning provided by the present application achieves the most advanced performance, highlighting its effectiveness in capturing the complexity of classroom interactions.

[0033] In recent years, artificial intelligence technology has made significant progress in behavior recognition, automatic translation, autonomous driving and many other fields, greatly facilitating the rapid and accurate extraction of information from a large amount of video, image and text data. In the field of education, classroom interaction behavior is an important indicator for evaluating teaching quality, providing a basis for accurately grasping student needs and adjusting teaching strategies. However, existing manual assessment methods are time-consuming and labor-intensive when dealing with large amounts of data. To reduce the workload of teachers analyzing video interactions and continuously monitor classroom interactions, the present application uses artificial intelligence technology to automate the process.

[0034] Reference Figure 1 , Figure 1 is a model reasoning space-time confusion diagram provided by the related technology of the present application, which includes (a) the tarining example and (b) the test example, the true situation (GT, Ground Truth) and the predicted situation (Predict).

[0035] Unlike object detection, which relies only on the visual appearance of the object, human behavior recognition requires the comprehensive analysis of the spatiotemporal associations between individuals, objects, and other people. This process is susceptible to the influence of data distribution, which can lead to the establishment of false spatiotemporal associations. For example, although "asking a question" is logically related to "answering a question" and should be able to predict the "answering a question" behavior based on "asking a question", if the frequency of "discussion" behavior in the dataset is much higher than "answering a question", such as Figure 1 (a), a false correlation between "asking a question" and "discussion" may be directly formed during the training process. Therefore, as shown in Figure 1 (b), during model inference, even if the "asking a question" behavior is correctly identified, the model may misjudge the subsequent behavior as "discussion". To address the problem of data imbalance, common methods include re-balancing training, hard example mining, and transfer learning. Re-balancing training improves the data distribution between classes through resampling or weighting; hard example mining selects samples with high diversity and high loss for training to optimize the model; and transfer learning reduces the impact of data imbalance through feature transfer. Although these strategies have achieved certain success, they mainly simulate the statistical relationship between data and labels, compensate for the related problems caused by data imbalance, but do not fundamentally solve the potential causal relationship between the two.

[0036] With the continuous development of causal inference theory, its ability to improve estimation accuracy and reduce bias has been widely recognized. This theory has begun to be applied in the field of computer vision, by capturing and eliminating confounding factors that are equally important in causing false correlations, such as spatial context information, which interferes with model predictions, thereby revealing the causal relationship between data and labels. In instructional video analysis, the interaction between behaviors involves both spatial and temporal dimensions, both of which can affect the establishment of correct spatiotemporal associations. Although existing techniques attempt to eliminate the influence of these interactions, they are usually limited to spatial factors or ignore the heterogeneity of spatial and temporal dimensions. In addition, due to the particularity of teaching activities, the dependency relationships between different behaviors also show significant differences, such as the interaction frequency of "asking a question-answering a question" is usually higher than "asking a question-accepting", which indicates that it cannot be simply assumed that the occurrence probability of all confounding factors is consistent.

[0037] There are still some problems in classroom interactive behavior recognition. Existing classroom behavior datasets usually only focus on the independent behavior of a specific participant (teacher / student), such as raising hands or sleeping, ignoring the dynamic nature of interaction, which negatively affects the evaluation of interaction quality. To solve these problems, the invention adopts a visual interactive behavior dataset (BNU-SVIBD) and proposes a classroom interactive behavior recognition method based on causal reasoning (CausalCIBR). BNU-SVIBD provides interactive sequences composed of various interactive behaviors and their durations occurring in a specific teaching environment. The sequence shows the dynamic process of behavior in interactive events, providing data support for intelligent evaluation of interaction quality. The visual interactive behavior dataset is the first relatively complete classroom interactive behavior dataset based on real classroom scenarios. Considering the heterogeneity of the spatio-temporal dimension, the classroom interactive behavior recognition method based on causal reasoning first obtains the average spatio-temporal dependence features of different behavior pairs through pre-training; then integrates these features into the model as decoupled spatio-temporal confounders. Second, the probability of the continuous occurrence of two behaviors collected from existing classroom interaction data is used as the prior probability of the confounder. Finally, causal intervention is performed on these confounders, and multi-scale gated feedforward is used to significantly improve the robustness of behavior recognition. In addition, PolyLoss (a new loss function framework proposed in deep learning and neural network training, which decomposes commonly used classification loss functions into a series of weighted polynomial bases through Taylor expansion. This framework allows the importance of different polynomial bases to be adjusted according to different tasks and datasets) is introduced to adjust the weight of the polynomial basis in the loss function according to the needs of the interaction sequence and behavior recognition task.

[0038] Current action recognition techniques are mainly based on convolutional neural networks (CNN) and Transformers. CNN-based action recognition methods usually use 2D / 3D CNN structures to model the spatio-temporal representation of actions. For example, I3D backbone network is used to extract RGB image features, and spatio-temporal consistency regularization (including temporal coherence and gradient smoothness) is used to improve the accuracy of action detection. 2D CNN is used to extract appearance and pose features from video and pose sequences, and an asynchronous fusion strategy is used in the time and spatial dimensions to identify actions. With the successful application of Transformers in natural language processing, this technology is increasingly used in video processing, strengthening the modeling of long-term dependencies. For example, a modality-specific time interval is used to encode the Transformer encoder to identify continuous actions in videos. A bottleneck cross-attention mechanism is proposed, enabling spatio-temporal expert models to exchange information and make collaborative predictions, thereby improving the performance of action recognition.

[0039] Although these methods have made progress in constructing spatio-temporal association representation of actions, they often ignore the potential negative impact of data imbalance on spatio-temporal association construction.

[0040] However, these methods often only eliminate the influence of spatial context information or treat spatio-temporal features as confounding factors, ignoring the heterogeneity of video data in the spatial and temporal dimensions. In addition, these studies usually assume that the occurrence probability of all confounding factors is the same, ignoring the diversity of dependency relationships between classroom behaviors under teaching rules, which leads to differences in the likelihood of different context information. The present invention takes this into account, considering both spatial and temporal confounding factors as interference factors in model prediction, and using the collected probability of the continuous occurrence of the two behaviors as the prior distribution of the confounding factors, to more accurately obtain the relationship between individual behaviors through causal intervention.

[0041] Alternatively, the classroom interaction behavior recognition method based on causal reasoning of the embodiments of the present application can be executed by a server, or by a terminal device, or by both a server and a terminal device. Taking the classroom interaction behavior recognition method based on causal reasoning in the embodiments as an example, the method is executed by a server.

[0042] Figure 2 is a flowchart of the classroom interaction behavior recognition method based on causal reasoning provided by the present invention, as shown in Figure 2 The method comprises the following steps.

[0043] Step 201, obtaining an interactive video input by a user, wherein the interactive video comprises individual behaviors in a classroom scene, and the individual behaviors comprise teacher behaviors and student behaviors.

[0044] In the embodiments of the present invention, the interactive video comprising a plurality of different individual behaviors in a classroom scene is obtained from a source file (such as a local file, online storage or real-time camera input) provided by a user. Here, the video content of the interactive video covers the classroom scene and contains teacher behaviors and student behaviors.

[0045] Step 202, inputting the interactive video into a classroom interaction behavior recognition model to obtain the recognition results of the interactive sequence and the individual behaviors output by the classroom interaction behavior recognition model; the interactive sequence is used to represent the alternating mode or interactive mode between the teacher behaviors and the student behaviors.

[0046] The classroom interaction behavior recognition model is trained based on the interactive video samples, the real class label of the interactive sequence corresponding to the interactive video samples, and the real class label of the individual behaviors corresponding to the interactive video samples.

[0047] In the embodiment of the present application, the pre-trained classroom interaction behavior recognition model is used to recognize the interaction behavior and individual behavior in the video, that is, the interaction video obtained in step 201 is input into the classroom interaction behavior recognition model; the classroom interaction behavior recognition model outputs the recognition result of the interaction sequence and the recognition result of the individual behavior.

[0048] The interaction sequence refers to the behavior sequence of the interaction between the teacher and the student in a specific teaching activity, including the behavior alternation model or interaction mode between the teacher and the student. For example, including the teacher's teaching and the student's listening, the teacher's questioning and the student's answering, the teacher's guidance and the student's execution, etc. These behavior sequences reflect the interaction dynamics between the teacher and the student in the classroom.

[0049] The individual behavior includes the teacher behavior and the student behavior, wherein the teacher behavior can be blackboard writing, explanation, walking, etc. The student behavior can be listening, note taking, participating in discussion, etc.

[0050] In the embodiment of the present application, the classroom interaction behavior recognition model is pre-trained by a large number of interaction video samples (visual interaction behavior data set) to accurately recognize various interaction behaviors and individual behaviors.

[0051] In some embodiments, the output result is provided in the form of text, including detailed description of the interaction behavior and individual behavior and timestamp information.

[0052] Through the above steps of the embodiment of the present application, the interaction video input by the user is obtained, which includes the student behavior and the teacher behavior. The interaction video is the participant interaction in the classroom scene, which ensures the pertinence and relevance of the data and avoids the interference of irrelevant information; the interaction video is input into the classroom interaction behavior recognition model, and the individual behavior (including the teacher behavior and the student behavior) in the classroom scene and the interaction sequence composed of these behaviors can be accurately recognized by the classroom interaction behavior recognition model. Therefore, the model not only recognizes the individual behavior, but also recognizes the interaction behavior and its duration. By recognizing the interaction sequence, the correlation and causality between different behaviors can be analyzed, which helps to deeply understand the classroom interaction mode. In addition, the classroom interaction behavior model can learn the accurate features of the classroom interaction behavior by training a large number of interaction video samples and their real class labels, thereby improving the recognition accuracy.

[0053] According to the classroom interaction behavior recognition method based on causal reasoning provided by the present application, the classroom interaction behavior recognition model includes an individual behavior capturing module, a causal relationship constructing module, a first full connection layer, a maximum pooling layer and a second full connection layer. The interaction video is input into the classroom interaction behavior recognition model to obtain the recognition result of the interaction sequence and the recognition result of the individual behavior output by the classroom interaction behavior recognition model, including:

[0054] The individual behavior capturing module is used for feature recognition of the interactive video, so as to obtain teacher behavior features and student behavior features;

[0055] The causal relation construction module is used for constructing the space-time correlation features between the teacher behavior features and the student behavior features;

[0056] Feature fusion is performed based on the teacher behavior features, the student behavior features and the space-time correlation features, so as to obtain fusion features;

[0057] The fusion features are subjected to causal intervention processing, so as to obtain causal features, wherein the causal intervention processing is used for eliminating false correlation of the fusion features;

[0058] The causal features are classified through the first fully connected layer, so as to obtain the recognition result of the individual behavior;

[0059] The causal features are classified through the maximum pooling layer and the second fully connected layer, so as to obtain the recognition result of the interactive sequence.

[0060] Reference Figure 3 , Figure 3 is the structural block diagram of the classroom interactive behavior recognition model based on causal reasoning provided by the application, wherein the input interactive video (a series of video frames), the individual behavior capturing module (IBC, Individual Behavior Capturing), the causal relation construction module (CRC, Causal Relation Construction), the maximum pooling (Maxpool), the fully connected (FC, Fully Connected), the individual behavior classification, that is, the recognition result of the individual behavior (Individual Classification) and the interactive behavior classification, that is, the recognition result of the interactive sequence (GroupClassification) are included.

[0061] The individual behavior capturing module includes a convolutional neural network (CNN), a RoIAlign (an improved RoI pooling method) and a fully connected (FC, Fully Connected); the causal relation construction module includes a group behavior recognition module (Construct Actor Relation Gragh) and a causal intervention module (Causal Intervention).

[0062] The input interactive video , represents the interactive video, represents a real number set, represents the frame number of the interactive video, and respectively represent the height and width of each video frame.

[0063] Inception-v3 and RoIAlign are used in the Individual Behavior Capture (IBC) module to extract the behavior features of teachers and students from multiple frames.

[0064] Here, Inception-v3: a deep convolutional neural network for image feature extraction. RoIAlign: a method for accurately extracting features of regions of interest (RoI) from feature maps, commonly used in object detection and segmentation tasks.

[0065] Then, the feature dimensions of the extracted behavior features are adjusted through a fully connected layer (FC) to obtain the behavior representation where, represents the behavior representation, represents the total number of bounding boxes detected within frames, is the feature dimension of each bounding box.

[0066] In the Causal Relationship Construction (CRC) module, a graph convolutional network (GCN) is used to construct a relationship graph based on these behavior standards to further obtain the spatiotemporal correlation features of student and teacher behavior where, represents the spatiotemporal correlation feature, represents the total number of bounding boxes detected within frames, is the feature dimension of each bounding box. To eliminate the confounding effect of spatiotemporal correlation features, causal intervention is implemented, where causal intervention processing is used to eliminate false correlations in fused features.

[0067] For example, based on structural equation modeling (SEM) or deep learning-based causal discovery algorithms. By simulating interventions (such as do operators) to observe the impact of changes in specific behavior features on other features, false correlations that are not caused by causal relationships are identified and eliminated.

[0068] Subsequently, the spatiotemporal correlation features are fused with the original behavior features (i.e., teacher behavior features, student behavior features) to obtain causal features after causal intervention processing, and through a fully connected layer (i.e., the first fully connected layer) and maxpool (maxpooling layer) and fully connected layer (second fully connected layer), individual behavior and interaction sequences of teachers and students are classified respectively.

[0069] For example, the causal features are classified by a first fully connected layer to obtain the recognition result of individual behaviors; that is, the fusion features are input into a fully connected layer, and a probability distribution of each individual behavior class is output by a softmax function.

[0070] The causal features are classified by a maximum pooling layer and a second fully connected layer to obtain the recognition result of interaction sequences; the maximum pooling layer is applied to reduce the dimension of the features and retain the most important information, and then the processed features are input into the second fully connected layer for classification. The probability distribution of the interaction sequence class is also output by a softmax function.

[0071] By combining individual behavior capture, causal relationship construction, causal intervention processing, and feature fusion and classification, the embodiments of the present application realize the recognition of the behaviors of teachers and students and the interaction sequences between them from interaction videos. Not only the behavior features themselves are considered, but also the causal relationships between behaviors are deeply analyzed, thereby improving the accuracy and robustness of the recognition.

[0072] Reference Figure 4 , Figure 4 is a schematic diagram of a framework for constructing a participant relationship graph provided by the present application.

[0073] In order to capture different types of teacher-student relationship information, the classroom interaction behavior recognition model further includes an interaction sequence dynamic module.

[0074] In the interaction sequence dynamic module (ISD module), a role relationship graph is constructed. The construction process of each graph is as shown in Figure 4 . First, in the ISD module, two linear transformations are performed on the spatiotemporal correlation features of all individuals, where represents the spatiotemporal correlation features, represents the total number of bounding boxes detected in frames, and is the feature dimension of each bounding box. Then, matrix multiplication is performed to obtain the relationship features between students and teachers; according to whether the Euclidean distance of the relationship features is within a certain range , the relationship features are shielded to improve the classification accuracy. Then, the masked features and the original relationship features are subjected to matrix multiplication and linear transformation operations; finally, the i-th role relationship graph is obtained.

[0075] According to the classroom interaction behavior recognition method based on causal reasoning provided by the present application, the fusion features are subjected to causal intervention processing to obtain causal features, including:

[0076] The average spatial context information between each pair of behavior categories in a single frame of the interactive video is obtained based on convolution of different scales;

[0077] Based on the average spatial context information, a spatial matrix is constructed, wherein the dimension of the spatial matrix is the product of the total number of pairs of behavior categories and the feature dimension of the spatial context of each pair of behavior categories;

[0078] The average temporal context information between each pair of behavior categories in the interactive video is obtained based on convolution of different time steps;

[0079] Based on the average temporal context information, a temporal matrix is constructed, wherein the dimension of the temporal matrix is the product of the total number of pairs of behavior categories and the feature dimension of the temporal context of each pair of behavior categories;

[0080] The spatial confusion dictionary corresponding to the spatial matrix and the time sequence confusion dictionary corresponding to the temporal matrix are determined;

[0081] Based on the inner product and matrix multiplication of the spatio-temporal correlation features in the fusion features and the spatial confusion dictionary and the time sequence confusion dictionary, an expected value is obtained;

[0082] Based on the spatio-temporal correlation features in the fusion features and the expected value, splicing and multi-scale gated feedforward processing are performed to obtain causal features.

[0083] Reference Figure 5 , Figure 5 is a schematic diagram of causal intervention provided by the present application, which includes (a) a confusion diagram (Confounded) and (b) a deconstruction diagram (Deconfounded).

[0084] In the embodiments of the present application, how data imbalance confuses spatio-temporal correlation and leads to performance degradation in recognition is discussed. In order to analyze in depth how spatio-temporal correlation is misled by confusion factors, a structural causal model (SCM) is constructed, which depicts the causal relationship between spatio-temporal correlation features X, temporal confusion factors , spatial confusion factors and target category labels (i.e. causal features) Y, as shown in Figure 5 (a). In this model, the arrows represent the causal relationship between variables.

[0085] Causal intervention in classroom interactive behavior recognition . In order to capture the true causal effect: , the backdoor paths and are blocked by cutting off and at the same time.

[0086] (1) and The temporal-spatial confounding here refers to the spatial context information determined by the data collection process and temporal context information distribution. Once collected, and determine what actions affect the spatio-temporal association features in the spatial and temporal dimensions, respectively.

[0087] For example, since "asking questions" and "taking notes" often occur simultaneously in the dataset, the model may construct a "question- note-taking" spatio-temporal association feature.

[0088] (2) and : Such spatial and temporal dimension information also affects the classifier's understanding of label prediction.

[0089] For example, since "asking questions" and "taking notes" often occur simultaneously in the dataset, the model will identify the "note-taking" label based on the context information "asking questions".

[0090] (3) : The general process of behavior recognition uses spatio-temporal association features X to obtain prediction results Y. Therefore, in the case of unbalanced distribution of the dataset, the model tends to learn the spurious correlation between X and Y caused by and , that is, overuse the co-occurrence between temporal / spatial context and class labels, leading to the learning of biased spatio-temporal association representations.

[0091] Inspired by the successful application of causal inference in the field of computer vision, the embodiments of the present application eliminate spurious correlations through causal intervention. Generally, the association of actions in instructional videos is influenced by spatial and temporal factors. However, existing research usually only eliminates the influence of spatial context information, and some research directly considers spatio-temporal features as confounding factors, ignoring the heterogeneity of the spatial and temporal dimensions of the video itself.

[0092] To solve this problem, as shown in (b) in Figure 5 , both spatial confounding factors and temporal confounding factors are captured, and the causal intervention is used to replace the conventional training objective . Here, operation cuts off the causal path of and , so that these confounding factors no longer interfere with the causal prediction path from X to Y, effectively eliminating spurious correlations. Causal intervention can be represented by the following formula:

[0093]

[0094] where, denotes the probability distribution of the variable under the intervention , denotes the probability, denotes the spatial confounder, denotes the temporal confounder, and X denotes the spatio-temporal associated feature, denotes the target class label (i.e., causal feature).

[0095] Given the complexity of the temporal / spatial context information in real data, we first utilize the multi-scale convolution technique to obtain the average spatial context information between each pair of classes in a single frame to construct the spatial matrix where, denotes the total number of action class pairs, denotes the feature dimension of the spatial context for each class pair.

[0096] Similarly, we use different time step convolutions to obtain the average temporal context information between each pair of classes to construct the temporal matrix where, denotes the feature dimension of the temporal context for each class pair. Considering the diversity of behavior dependency relationships in teaching activities, we assign weights to the entries in and by integrating and computing the average correlation between teacher and student behaviors.

[0097] Subsequently, the model is trained to learn two linear projections (spatial linear projection and temporal linear and to transform and into spatial and temporal confounding dictionaries and , i.e. , . Therefore, the causal intervention can be approximated by the NWGM (Normalized Weighted Geometric Mean) as follows:

[0098]

[0099] where, denotes the probability distribution of the variable under the intervention , denotes the probability, denotes the spatial confounder (i.e., spatial confounding dictionary), denotes the temporal confounder (i.e., temporal confounding dictionary), and X denotes the spatio-temporal associated feature, denotes the target category label (i.e., causal feature); for mapping a real-valued vector to a probability distribution, denote the expectation (or mean) of , , . An FC layer is usually used.

[0100] Referring to Figure 6 , Figure 6 is a structural diagram of a multi-scale gated feed-forward module provided by the present application, which comprises: a 1*1 PixelUnshuffle operation (converting a low-resolution feature map into a high-resolution image through a specific arrangement), a 3*3 depth convolution (DWConv, Depthwise Convolution), a multi-scale dilated convolution (Multi-scaleDilatedConv), and a 1*1 PixelShuffle operation (PixelShuffle learns the mapping relationship between a low-resolution image and a high-resolution image, and up-samples the low-resolution image into a high-resolution image).

[0101] In order to further enhance the ability of the causal representation to extract multi-scale information, the present application proposes a multi-scale gated feed-forward (MGF) module to replace the FC layer. As shown in Figure 6 , first, 1*1 convolution and pixel up-sampling are applied to the causal representation to extract high-resolution semantic information. Next, the high-resolution information is fed into two parallel convolution paths. One path uses a 3*3 depth separable convolution, while the other path uses a multi-scale dilated convolution. After that, the features from the two paths undergo a blurred activation function and element-wise multiplication. Finally, pixel down-sampling and 1*1 convolution are applied to the fused representation (fused features).

[0102] Referring to Figure 7 , Figure 7 is a flowchart of a causal intervention module provided by the present application, which comprises input (X), temporal confounding dictionary (Temporal confounding dictionary), spatial confounding dictionary (Spatial confounding dictionary), inner product (Inner product), Softmax function, residual connection and layer normalization (Add&LayerNorm), multi-scale gated feed-forward module and activation function (MGF&Softmax), and output representation Y (i.e., causal feature) that is not disturbed by temporal and spatial confounding factors.

[0103] The causal intervention module flowchart is as shown in Figure 7As shown. Specifically, the input feature X is subjected to inner product and matrix multiplication with the time confusion dictionary to obtain a corresponding first expectation value. Similarly, a second expectation value of the input feature and the spatial confusion dictionary is obtained. Then, using the splicing and MGF module, the two expectation values are fused to generate a representation Y (i.e. causal feature) that is not disturbed by the time and space mixed factors.

[0104] Through the embodiment of the present application, the introduction of the spatial confusion dictionary and the time confusion dictionary can efficiently fuse the space-time correlation features with the existing feature information, thereby extracting more representative features; the multi-scale gated feedforward network is used to further process the spliced features, which can fully utilize the feature information of different scales, and then obtain the causal features that are not disturbed by the time and space mixed factors.

[0105] According to the classroom interaction behavior recognition method based on causal reasoning provided by the present application, before the interaction video is input into the classroom interaction behavior recognition model to obtain the recognition result of the interaction sequence and the recognition result of the individual behavior output by the classroom interaction behavior recognition model, the above method further comprises:

[0106] Obtain an interaction video sample, wherein the interaction video sample carries a real class label of the interaction sequence and a real class label of the individual behavior;

[0107] Input the interaction video sample into a preset classroom interaction behavior recognition model to obtain a predicted class label of the interaction sequence and a predicted class label of the individual behavior output by the preset classroom interaction behavior recognition model;

[0108] Based on a target loss function, according to the real class label of the interaction sequence and the predicted class label of the interaction sequence, and the real class label of the individual behavior and the predicted class label of the individual behavior, the preset classroom interaction behavior recognition model is updated to obtain a trained classroom interaction behavior recognition model.

[0109] In the embodiment of the present application, the interaction video sample records the interaction behavior in the classroom environment, including the interaction between students and teachers, the interaction between students, etc.; each interaction video sample is attached with two types of labels.

[0110] That is, the real class label of the interaction sequence: describes the behavior category occurring in the entire interaction sequence, such as "teacher asks a question", "student answers" and the like. The real class label of the individual behavior: details the specific behavior of each individual (such as a student or a teacher) in the video, such as "raising hand", "standing" and the like.

[0111] The classroom interaction behavior recognition model is used for recognizing interaction behaviors and individual behaviors in a classroom.

[0112] The target loss function is a function for measuring the difference between the predicted result and the actual result of the model, including two parts: the interaction sequence loss (measuring the accuracy of the model in predicting the behavior category of the interaction sequence) and the individual behavior loss (measuring the accuracy of the model in predicting the behavior category of the individual).

[0113] According to the calculation result of the target loss function, the weight and bias parameters of the model are updated using the back propagation algorithm to reduce the difference between the predicted result and the actual result; after multiple iterations of training, the model will gradually converge and be able to provide accurate prediction results on new interaction video samples, obtaining the trained classroom interaction behavior recognition model.

[0114] Through the training process described above, a model capable of accurately recognizing classroom interaction behaviors and individual behaviors can be obtained, which helps to improve the accuracy of classroom analysis.

[0115] According to the classroom interaction behavior recognition method based on causal reasoning provided by the present application, the target loss function refers to the following formula:

[0116] ;

[0117] Among them, the target loss function is represented by L, the first cross-entropy loss is represented by L1, the real category label of the interaction sequence is represented by y, the predicted category label of the interaction sequence is represented by y, the first hyperparameter is represented by a, the prediction accuracy for the interaction sequence is represented by p, the second cross-entropy loss is represented by L2, the real category label of the individual behavior is represented by y, the predicted category label of the individual behavior is represented by y, the second hyperparameter is represented by b, the prediction accuracy for the individual behavior is represented by p.

[0118] In the embodiment of the present application, in order to improve the accuracy of model training, ployloss is introduced for end-to-end training. This strategy adjusts the weight of the polynomial base in the cross-entropy loss function by introducing two hyperparameters, making it more suitable for complex classroom interaction behavior recognition scenarios. The specific loss function can refer to the above formula.

[0119] Here, PolyLoss is a new loss function framework that unifies Cross-entropy Loss and Focal Loss, specifically, PolyLoss considers and designs the loss function as a linear combination of polynomial functions, for example, it can decompose the commonly used classification loss function into a series of weighted polynomial bases.

[0120] Through the embodiment of the application, the weights and bias parameters of the model can be updated using the back propagation algorithm according to the calculation result of the target loss function, so as to reduce the difference between the prediction result and the actual result.

[0121] According to the classroom interaction behavior recognition method based on causal reasoning provided by the application, the interaction video sample includes a plurality of individual behaviors and a plurality of interaction sequences.

[0122] The plurality of individual behaviors include acceptance, rejection, questioning, discussion, active conversation, response, note taking or practice, teaching and guidance.

[0123] The interaction behaviors of the plurality of interaction sequences include teacher teaching and student listening, teacher instruction and student execution, question and answer, and group discussion.

[0124] Reference Figure 8 , Figure 8 is a schematic diagram of the actual scene from the perspective of the student (a) and the teacher (b) provided by the application.

[0125] It should be noted that the interaction video sample is derived from the visual interaction behavior dataset.

[0126] According to the theory of the classic classroom interaction observation and analysis system (Flanders interaction analysis system, FIAS), through the integration of the characteristics of data collection, nine behaviors that can be captured through body movements and facial expressions are summarized from a large number of behaviors.

[0127] These behaviors include acceptance, rejection, questioning, discussion, active conversation, response, note taking / practice, teaching and guidance.

[0128] The dataset provides interaction sequences composed of multiple interaction behaviors and interaction durations occurring in a teaching scene, showing the dynamic process of behaviors in a specific teaching situation, rather than isolated, irrelevant actions. They cover a variety of teaching situations such as teacher teaching-student listening, teacher ordering-student execution, question and answer, and group discussion, among which group discussion is further divided into student discussion, student-teacher group discussion and teacher discussion sub-sequences.

[0129] To restore the real classroom scene and improve the quality of data collection, the collection environment is selected in the real classroom. A high-definition camera is installed at the front (top right corner) and back of each classroom, respectively, corresponding to the student and teacher's perspective. Ten videos are randomly selected to represent discussions and teaching equally to obtain sufficient sequences and behavior samples. These videos only clearly record the teacher's voice, not the student's voice. Each video is 45 minutes long and has a resolution of 1920 x 1080. During the annotation process, the entire teaching process is first divided based on the four interaction sequences described above. Second, they manually annotate each group in the discussion sequence based on the three interaction sub-sequences described above to clearly define the start time, end time, and participants of each interaction process.

[0130] Next, the interaction behavior in each interaction process is manually annotated. Finally, 114 interaction sequence samples, 332 sub-sequence samples, and 4838 interaction behavior samples are obtained. According to the 3:1 ratio, these samples are divided into training data sets and test sets to form the visual interaction behavior data set (BNU-SVIBD). This is a relatively complete classroom interaction behavior data set based on real classroom videos. It is worth noting that the data exhibits a clear long-tail distribution. Specifically, the most frequent behavior is discussion, with 2481 occurrences, followed by note-taking / exercise, with 1239 occurrences, while refusal, questioning, speaking out, responding, and guidance all occur less than 100 times, with speaking out occurring only 9 times.

[0131] Reference Figure 9 , Figure 9 is a qualitative analysis diagram on the visual interaction behavior data set provided by the present application, which includes: a series of video frames, ground truth (GT), recognition results without using causal intervention (w / o CI), and recognition results using causal intervention (Ours).

[0132] To further demonstrate the effectiveness of causal intervention (causal intervention), visual analysis is performed on some example videos, comparing the recognition results using causal intervention (Ours) and without using causal intervention (w / o CI). By dividing the video by group and inputting it into the model for testing, the interaction sequence and behavior of the interactors can be specifically displayed, and the results are as follows: Figure 8As the data distribution of the visual interaction behavior dataset presents a long-tail characteristic, the model without using causal intervention is vulnerable to high-frequency categories such as 'discussion', and it is difficult to accurately construct the spatio-temporal correlation of 'question-answer' or 'discussion-acceptance'. In addition, the model without using causal intervention is also difficult to accurately capture the interaction sequence from the relationship between individual behaviors such as 'answer-note-taking'. The proposed method effectively identifies accurate classroom interaction sequences and behaviors from the relationship between individual behaviors by intervening in the weighted spatial and temporal confounding factors, overcoming the influence of the data distribution.

[0133] Through the above steps of the embodiment of the present application, a visual interaction behavior dataset is provided, and a classroom interaction behavior recognition method based on causal reasoning is proposed on this basis, which provides key data support and technical framework for in-depth analysis of classroom interaction quality. The visual interaction behavior dataset indicates 9 kinds of interaction behaviors and provides interaction sequence labels containing multiple behaviors and durations, showing the dynamic process of interaction. The framework eliminates the interference of time and space confounding factors on model prediction by identifying and processing these factors and assigning weights to these factors according to the teaching law, thereby providing unbiased spatio-temporal representation. In order to further optimize the model learning process, the ployloss algorithm is introduced, which assigns different weights to the polynomial bases in the loss function of the interaction sequence and behavior. Experimental results show that this method has achieved significant results in classroom interaction sequence and behavior recognition.

[0134] The classroom interaction behavior recognition device based on causal reasoning provided by the present application is described below. The classroom interaction behavior recognition device based on causal reasoning described below can be mutually corresponding with the classroom interaction behavior recognition method based on causal reasoning described above.

[0135] Reference Figure 10 , Figure 10 is a structural schematic diagram of the classroom interaction behavior recognition system based on causal reasoning provided by the present application.

[0136] The acquisition module 1001 is configured to acquire an interaction video input by a user, wherein the interaction video includes individual behaviors in a classroom scene, and the individual behaviors include teacher behaviors and student behaviors.

[0137] The recognition module 1002 is configured to input the interaction video into a classroom interaction behavior recognition model to obtain recognition results of an interaction sequence and recognition results of individual behaviors output by the classroom interaction behavior recognition model; the interaction sequence is used to represent an alternating mode or an interaction mode between the teacher behaviors and the student behaviors.

[0138] The classroom interaction behavior recognition model is trained based on an interaction video sample, a real class label of an interaction sequence corresponding to the interaction video sample, and a real class label of individual behavior corresponding to the interaction video sample.

[0139] Specifically, the classroom interaction behavior recognition system based on causal reasoning provided by the present application can realize all method steps implemented by the classroom interaction behavior recognition method embodiment based on causal reasoning, and achieve the same technical effects. Here, the same parts and beneficial effects of the method embodiment in this embodiment will not be described in detail.

[0140] Figure 11 is a schematic diagram of the physical structure of the electronic device provided by the present application, as shown in Figure 11 The electronic device can include a processor 1111, a communications interface 1120, a memory 1130, and a communications bus 1140, wherein the processor 1111, the communications interface 1120, and the memory 1130 communicate with each other through the communications bus 1140. The processor 1111 can invoke the logical instructions in the memory 1130 to execute the classroom interaction behavior recognition method based on causal reasoning, which includes: obtaining an interaction video input by a user, wherein the interaction video includes individual behavior in a classroom scene, and the individual behavior includes teacher behavior and student behavior; inputting the interaction video into a classroom interaction behavior recognition model to obtain the recognition result of the interaction sequence and the recognition result of the individual behavior output by the classroom interaction behavior recognition model; the interaction sequence is used to represent the alternating mode or interactive mode between the teacher behavior and the student behavior; wherein the classroom interaction behavior recognition model is trained based on an interaction video sample, a real class label of an interaction sequence corresponding to the interaction video sample, and a real class label of individual behavior corresponding to the interaction video sample.

[0141] Further, the logic instructions in the memory 1130 described above can be implemented in the form of software functional units and sold or used as standalone products, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partially contribute to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0142] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the classroom interaction behavior recognition method based on causal reasoning provided by the above-mentioned methods. The method comprises: obtaining an interaction video input by a user, wherein the interaction video comprises individual behaviors in a classroom scene, and the individual behaviors comprise teacher behaviors and student behaviors; inputting the interaction video into a classroom interaction behavior recognition model to obtain an identification result of an interaction sequence output by the classroom interaction behavior recognition model and an identification result of the individual behaviors; the interaction sequence is used to represent an alternating mode or an interactive mode between the teacher behaviors and the student behaviors; and the classroom interaction behavior recognition model is trained based on an interaction video sample, a real class label of an interaction sequence corresponding to the interaction video sample, and a real class label of individual behaviors corresponding to the interaction video sample.

[0143] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the classroom interaction behavior recognition method based on causal reasoning provided by the above-mentioned methods. The method comprises: obtaining an interaction video input by a user, wherein the interaction video comprises individual behaviors in a classroom scene, and the individual behaviors comprise teacher behaviors and student behaviors; inputting the interaction video into a classroom interaction behavior recognition model to obtain an identification result of an interaction sequence output by the classroom interaction behavior recognition model and an identification result of the individual behaviors; the interaction sequence is used to represent an alternating mode or an interactive mode between the teacher behaviors and the student behaviors; and the classroom interaction behavior recognition model is trained based on an interaction video sample, a real class label of an interaction sequence corresponding to the interaction video sample, and a real class label of individual behaviors corresponding to the interaction video sample.

[0144] The apparatus embodiments described above are merely illustrative, wherein the units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0145] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary universal hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0146] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for recognizing classroom interactive behaviors based on causal reasoning, characterized in that, include: The system acquires interactive videos input by the user, wherein the interactive videos include individual behaviors in a classroom setting, and the individual behaviors include teacher behaviors and student behaviors. The interactive video is input into the classroom interactive behavior recognition model to obtain the recognition results of the interactive sequence and the recognition results of individual behaviors output by the classroom interactive behavior recognition model; the interactive sequence is used to represent the alternation pattern or interaction pattern between the teacher's behavior and the student's behavior. The classroom interaction behavior recognition model is trained based on interactive video samples, the real category labels of the interaction sequences corresponding to the interactive video samples, and the real category labels of the individual behaviors corresponding to the interactive video samples. The classroom interaction behavior recognition model includes: an individual behavior capture module, a causal relationship construction module, a first fully connected layer, a max pooling layer, and a second fully connected layer. The interactive video is input into the classroom interaction behavior recognition model to obtain the recognition results of the interaction sequence and the recognition results of individual behaviors output by the model, including: The individual behavior capture module performs feature recognition on the interactive video to obtain teacher behavior characteristics and student behavior characteristics. The spatiotemporal correlation features between the teacher's behavioral characteristics and the student's behavioral characteristics are constructed using the causal relationship construction module. Based on the teacher behavior characteristics, the student behavior characteristics, and the spatiotemporal correlation characteristics, feature fusion is performed to obtain fused features; The fused features are subjected to causal intervention processing to obtain causal features, wherein the causal intervention processing is used to eliminate spurious correlations in the fused features; The causal features are classified by the first fully connected layer to obtain the identification results of individual behaviors; The causal features are classified using the max pooling layer and the second fully connected layer to obtain the identification results of the interaction sequence.

2. The classroom interaction behavior recognition method based on causal reasoning according to claim 1, characterized in that, The causal intervention processing of the fused features to obtain causal features includes: The average spatial context information between each action category pair in a single frame of the interactive video is obtained based on convolutions at different scales. Based on the average spatial context information, a spatial matrix is ​​constructed, wherein the dimension of the spatial matrix is ​​the product of the total number of behavior category pairs and the feature dimension of the spatial context of each behavior category pair; The average temporal context information between each pair of behavior categories in the interactive video is obtained based on convolutions with different time steps; Based on the average time context information, a time matrix is ​​constructed, wherein the dimension of the time matrix is ​​the product of the total number of behavior category pairs and the feature dimension of the time context of each behavior category pair; Determine the spatial confusion dictionary corresponding to the spatial matrix, and the temporal confusion dictionary corresponding to the temporal matrix; Based on the spatiotemporal correlation features in the fusion features, the inner product and matrix multiplication are performed with the spatial confusion dictionary and the temporal confusion dictionary to obtain the expected value; Based on the spatiotemporal correlation features in the fusion features and the expected value, causal features are obtained by splicing and multi-scale gating feedforward processing.

3. The classroom interaction behavior recognition method based on causal reasoning according to claim 1, characterized in that, Before inputting the interactive video into the classroom interaction behavior recognition model to obtain the recognition results of the interaction sequence and the recognition results of individual behaviors output by the classroom interaction behavior recognition model, the method further includes: Obtain interactive video samples, wherein the interactive video samples carry real category labels of the interaction sequence and real category labels of individual behaviors; The interactive video samples are input into a preset classroom interactive behavior recognition model to obtain the predicted category labels of the interactive sequences and the predicted category labels of individual behaviors output by the preset classroom interactive behavior recognition model. Based on the objective loss function, the preset classroom interaction behavior recognition model is updated according to the true category label and the predicted category label of the interaction sequence, as well as the true category label and the predicted category label of the individual behavior, to obtain the trained classroom interaction behavior recognition model.

4. The classroom interaction behavior recognition method based on causal reasoning according to claim 3, characterized in that, The target loss function is based on the following formula: ； in, Denotes the target loss function. This represents the first cross-entropy loss. This represents the true category label of the interaction sequence. This represents the predicted category label of the interaction sequence. Indicates the first hyperparameter. This represents the prediction accuracy for the stated interaction sequence. This represents the second cross-entropy loss. This indicates the true category label of the individual's behavior. The predicted category label representing the individual's behavior. This represents the second hyperparameter. This indicates the accuracy of the prediction for the individual's behavior.

5. The classroom interaction behavior recognition method based on causal reasoning according to claim 3, characterized in that, The interactive video samples include multiple individual behaviors and multiple interaction sequences; The aforementioned individual behaviors include: accepting, rejecting, asking questions, discussing, initiating conversations, responding, taking notes or doing exercises, lecturing, and instructing; The interactive behaviors of the multiple interactive sequences include: teacher lecturing and student listening, teacher instructions and student execution, question and answer, and group discussion.

6. A classroom interaction behavior recognition system based on causal reasoning, characterized in that, include: The acquisition module is used to acquire interactive videos input by the user, wherein the interactive videos include individual behaviors in a classroom setting, and the individual behaviors include teacher behaviors and student behaviors. The recognition module is used to input the interactive video into the classroom interactive behavior recognition model to obtain the recognition results of the interactive sequence and the recognition results of individual behaviors output by the classroom interactive behavior recognition model; the interactive sequence is used to represent the alternation pattern or interaction pattern between the teacher's behavior and the student's behavior. The classroom interaction behavior recognition model is trained based on interactive video samples, the real category labels of the interaction sequences corresponding to the interactive video samples, and the real category labels of the individual behaviors corresponding to the interactive video samples. The classroom interaction behavior recognition model includes: an individual behavior capture module, a causal relationship construction module, a first fully connected layer, a max pooling layer, and a second fully connected layer. The interactive video is input into the classroom interaction behavior recognition model to obtain the recognition results of the interaction sequence and the recognition results of individual behaviors output by the model, including: The individual behavior capture module performs feature recognition on the interactive video to obtain teacher behavior characteristics and student behavior characteristics. The spatiotemporal correlation features between the teacher's behavioral characteristics and the student's behavioral characteristics are constructed using the causal relationship construction module. Based on the teacher behavior characteristics, the student behavior characteristics, and the spatiotemporal correlation characteristics, feature fusion is performed to obtain fused features; The fused features are subjected to causal intervention processing to obtain causal features, wherein the causal intervention processing is used to eliminate spurious correlations in the fused features; The causal features are classified by the first fully connected layer to obtain the identification results of individual behaviors; The causal features are classified using the max pooling layer and the second fully connected layer to obtain the identification results of the interaction sequence.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the classroom interaction behavior recognition method based on causal reasoning as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the classroom interactive behavior recognition method based on causal reasoning as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the classroom interactive behavior recognition method based on causal reasoning as described in any one of claims 1 to 5.

Images