Double-stage classroom teacher teaching behavior detection method and device and medium

Abstract

The invention discloses a double-stage classroom teacher teaching behavior detection method and device, and a medium. The method comprises the following steps: receiving classroom video image data collected by a camera in real time; inputting the received classroom video image data into a teacher position detection model, and detecting the position of a teacher as a first-stage detection result; partial convolution operation in the teacher position detection model adopts frequency self-adaptive expansion convolution, and the frequency self-adaptive expansion convolution performs convolution operation on weighted features by using a self-adaptive expansion rate and a self-adaptive convolution kernel through realizing frequency selection to obtain an output feature layer; and determining a teacher position area in the video image data according to the first-stage detection result, and inputting the teacher position area and the classroom video image data into a teaching behavior recognition model for teaching behavior detection to obtain a behavior category of the teacher. The method has the advantages of being simple in implementation method, low in cost, high in detection precision, high in robustness and the like.

Description

Technical Field

[0001] The present invention relates to the field of smart education technology, and in particular to a method, device and medium for detecting teaching behavior of teachers in a double-stage classroom. Background Art

[0002] In traditional classroom teaching, teachers' classroom behavior evaluation is usually carried out by classroom observation, behavior comparison and other empirical description methods or by watching classroom videos, which leads to problems such as large workload, poor timeliness, high labor costs, strong subjectivity, and inability to achieve normalized evaluation. Using intelligent technology to detect teachers' classroom teaching behavior and then conduct process-based intelligent evaluation of classroom teaching behavior can help teachers improve their teaching behavior, which is of great significance to improving teaching quality.

[0003] With regard to the detection and identification of classroom teachers' teaching behaviors, the existing technology uses artificial intelligence technology to identify individual human body movements. However, in addition to teachers, there are students in the classroom. Directly identifying human body behaviors cannot distinguish the identities of students and teachers, which may result in students' behaviors being mistakenly identified as teachers' behaviors, causing interference with the detection of teachers' classroom teaching behaviors, resulting in low reliability of classroom teaching behavior evaluation. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a two-stage classroom teacher teaching behavior detection method, equipment and medium with simple implementation method, low cost, high detection accuracy and strong robustness.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is: A method for detecting teaching behavior of teachers in a two-stage classroom, comprising the following steps: Receive classroom video image data collected by the camera in real time; The received classroom video image data is input into the pre-trained teacher position detection model to detect the teacher's position as the first stage detection result; The teacher position detection model is a target detection model built based on the YOLOv (You Only Look Once) network, and some convolution operations in the YOLOv network use frequency adaptive dilated convolution FADConv, which performs Fourier transform on the feature map to obtain the corresponding frequency channel map, and decomposes it into multiple frequency channel maps according to the frequency, obtains the selection map of each frequency channel by convolution operation on the feature map, multiplies the selection map with the corresponding frequency channel map to obtain weighted features to achieve frequency selection, and performs convolution operation on the weighted features using an adaptive dilation rate and an adaptive convolution kernel to obtain an output feature layer; Based on the detection results of the first stage, the teacher's position area in the video image data is determined, and the determined teacher's position area and the classroom video image data are input into a pre-trained teaching behavior recognition model to perform teacher's teaching behavior detection, and the teacher's behavior category is obtained as the detection result output of the second stage.

[0006] Furthermore, the teacher position detection model is constructed based on the YOLOV9 network, the backbone network in the YOLOV9 network includes a RepNCSPELAN4 module and multiple detection heads, and the bottleneck layer in the RepNCSPELAN4 module and the convolution operation in the YOLOV9 initial feature module adopt the frequency adaptive dilated convolution FADConv.

[0007] Furthermore, the adaptive expansion rate is determined for each feature pixel by adopting convolution and Relu activation function, the convolution kernel parameters are decomposed into low-frequency convolution kernel parameters and high-frequency convolution kernel parameters, and the ratio of the high-frequency convolution kernel weight to the low-frequency convolution kernel weight is adjusted according to the input context to form the adaptive convolution kernel. When the average pooling operation is performed in the low-frequency convolution, the average is taken by summing the average pooling and the median pooling, and the channel weight is predicted by the first convolution layer conv1, the Relu activation function, the MLP layer, and the sigmod function layer in sequence.

[0008] Furthermore, the teacher position detection model also includes a feature fusion module for fusing feature maps, the feature fusion module includes an ASSP module and a CA module, the ASSP module realizes semantic feature extraction of different scales by adopting convolution, average pooling and dilated convolution with different expansion rates, the CA module fuses the feature maps extracted by the ASSP module through a residual module, and decomposes the channel attention into two one-dimensional feature encoding processes to aggregate features along two spatial directions respectively.

[0009] Furthermore, the teaching behavior recognition model is based on the Swin Transformer architecture as a feature extraction model. When the teaching behavior recognition model performs teaching behavior detection, the input image is preliminarily abstracted and divided into blocks using a convolutional neural network, and the feature maps after division are respectively subjected to feature extraction using the Swin Transformer architecture, and finally the detection result of the teaching behavior category is obtained. The W-MSA module of the Swin Transformer architecture and the MLP module in the SW-MSA module are implemented using the KAN (Kolmogorov-Arnold Networks) module, and the KAN module is formed by combining a spline curve and an MLP.

[0010] Furthermore, the representation structure of the KAN module is as follows:

[0011] in, , is the activation function , is a spline function, x represents the input feature, represents the KAN module output corresponding to the input feature x, L represents the number of network layers, represents the weight coefficient, Represents a composite operation of a function.

[0012] Furthermore, the loss function of the teacher position detection model is Including classification loss function And the position regression loss function , the calculation expression is:

[0013]

[0014]

[0015]

[0016] in, represents the weight coefficient, represents the class-weighted focal loss function for the i-th class person, is the class weight factor of the student or teacher, where , m represents the number of identity categories, is the number of positive samples of the i-th category of identity, is the predicted value of the i-th category person, is the true label value of the i-th category person, is the balance factor, is the weight factor, represents the bounding box loss, represents the distribution focal loss, , Represents the weight coefficient.

[0017] Furthermore, it also includes calculating a first indicator based on the first stage detection results and calculating a second indicator based on the second stage results, and evaluating the teacher's teaching behavior based on the first indicator and the second indicator. The first indicator includes the number of times the teacher is detected, and the second indicator includes the total number of times the teacher's teaching behavior is detected.

[0018] An electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0019] A computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0020] Compared with the prior art, the beneficial effect of the present invention lies in: starting from the perspective of fine-grained target recognition, the present invention adopts a two-stage detection method to realize teacher teaching behavior detection. First, the teacher position detection is realized by adopting a teacher position detection model constructed based on the YOLOv network, so that the teacher's position can be identified and located first. At the same time, some convolution operations in the network adopt frequency adaptive dilated convolution. The frequency selection, adaptive dilation rate and adaptive convolution kernel can achieve a balance between the local frequency effective bandwidth and the local receptive field of the feature map, so that at a relatively low computational cost, the model's ability to represent the local features of the feature map can be improved, so that character features can be obtained from a fine-grained perspective, and the distinction between teachers and students can be improved, thereby improving the recognition accuracy of teachers. After identifying the teacher's position, the teacher's behavior is detected using the identified teacher position, which can accurately distinguish the teacher's behavior category, reduce the interference of student behavior on detection, and improve the detection accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the implementation flow of the dual-stage classroom teacher teaching behavior detection method of Example 1 of the present invention.

[0022] Figure 2 It is a schematic diagram of the structural principle of the teacher position detection model adopted in Example 1 of the present invention.

[0023] Figure 3 It is a schematic diagram of the structural principle of the RepNCSPELAN4 module used in Example 1 of the present invention.

[0024] Figure 4 It is a schematic diagram of the structural principle of the RepNCSP module used in Example 1 of the present invention.

[0025] Figure 5 It is a schematic diagram of the structural principle of the FCBL module used in Example 1 of the present invention.

[0026] Figure 6 It is a schematic diagram of the structural principle of the CBL module used in Example 1 of the present invention.

[0027] Figure 7 It is a schematic diagram of the structural principle of the RepNBottleneck module implemented in Example 1 of the present invention.

[0028] Figure 8 It is a schematic diagram of the structural principle of the RepACconvN module implemented in Example 1 of the present invention.

[0029] Fig. 9 It is a schematic diagram of the structural principle of the ACconv convolution module training implemented in Example 1 of the present invention.

[0030] Fig.10 It is a schematic diagram of the structural principle when implementing ACconv convolution module reasoning in Example 1 of the present invention.

[0031] Fig.11 It is a schematic diagram of the structural principle of implementing frequency adaptive dilated convolution FADConv in Example 1 of the present invention.

[0032] Fig.12 It is a schematic diagram of the structural principle of the feature fusion module (ASSP+CA module) used in Example 1 of the present invention.

[0033] Fig.13 It is a schematic diagram of the processing flow for realizing teaching behavior recognition in Example 1 of the present invention.

[0034] Fig.14 It is a schematic diagram of the structural principle of the Swin Transformer architecture based on KAN in Example 1 of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0036] Students and senior teachers are two main types of people in the classroom. When identifying teacher-related teaching behaviors, students are interference factors, while when identifying student-related classroom behaviors, teachers are interference factors. Therefore, to achieve the accuracy and reliability of teacher classroom behavior detection, it is necessary to correctly distinguish the identities of students and teachers. Since the classroom is a relatively closed environment, there is basically no flow of people, only students and teachers, and there are fewer interference items. The teacher's position is generally located in the pedestrian area before and after the podium area and between desks. Drawing on the human eye to distinguish between teachers and students, it is usually based on their position, appearance and other features. From the perspective of fine-grained target recognition, the present invention adopts a two-stage detection method to realize teacher teaching behavior detection. First, the teacher position detection is realized by adopting a teacher position detection model constructed based on the YOLOv network, so that the teacher's position can be identified and located first. At the same time, some convolution operations in the network use frequency adaptive dilated convolution F. ADConv, by using frequency selection, adaptive dilation rate and adaptive convolution kernel, can achieve a balance between the effective bandwidth of the local frequency of the feature map and the local receptive field, so that at a lower computational cost, the model's ability to represent the local features of the feature map can be improved. Therefore, it can obtain character features from a fine-grained perspective, improve the distinction between teachers and students, and thus improve the accuracy of teacher recognition. After identifying the teacher's position, the teacher's behavior is detected using the identified teacher's position, which can accurately determine the teacher's behavior category, reduce the interference of student behavior on detection, and improve the accuracy and robustness of detection.

[0037] Embodiment 1: like Figure 1 As shown, the steps of the dual-stage classroom teacher teaching behavior detection method of this embodiment include: Step S01. Receive classroom video image data collected by the camera in real time.

[0038] Specifically, a camera can be arranged in the classroom to capture real-time video images of the teacher's teaching process, and the real-time captured classroom teaching video image frames can be uploaded to the data processor or control terminal, and the data processor or control terminal executes subsequent steps S02 and S03 to realize classroom teacher teaching behavior detection.

[0039] Step S02. Input the received classroom video image data into the pre-trained teacher position detection model to detect the teacher's position as the first stage detection result. The teacher position detection model is a target detection model built on the YOLOv network, and some convolution operations in the YOLOv network use frequency adaptive dilated convolution FADConv. Frequency adaptive dilated convolution FADConv decomposes the feature map into multiple frequency channels, obtains the selection map of each frequency channel through convolution operation, multiplies the selection map with the corresponding frequency channel to obtain weighted features to achieve frequency selection, and uses adaptive dilation rate and adaptive convolution kernel to perform convolution operation on the weighted features to obtain the output feature layer.

[0040] The purpose of teacher position detection is to correctly detect the teacher from the camera field of view to eliminate the interference of students in the field of view, thereby improving the accuracy of subsequent teacher teaching behavior detection. Since teachers and students belong to the same category with different identities, classroom teacher position detection can be attributed to fine-grained target detection. When designing the teacher position detection model, constructing a feature extraction and recognition network from a fine-grained perspective can improve the model's recognition accuracy for teachers and students.

[0041] In this embodiment, the teacher position detection model adopts a target detection model built based on the YOLOv network. The YOLOv series network divides the input image into a grid of a fixed size. Each grid unit is responsible for detecting the target in the area. If the center point of a target falls within a certain grid, the grid is responsible for predicting the bounding box and category of the target, and has efficient real-time processing capabilities. As an optional implementation, considering the balance between algorithm accuracy and real-time performance, the teacher position detection model can adopt a target detection model based on the YOLOV9 network, that is, the YOLOV9 network is used as the basic network architecture for teacher position detection. The YOLOV9 network further improves the accuracy of the detection task by introducing programmable gradient information (PGI) and generalized efficient layer aggregation network (GELAN).

[0042] Further, improvements and optimizations are made based on the YOLOV9 network, such as Figure 2As shown in the figure, the YOLOV9 network includes a backbone network and auxiliary branches. The backbone network includes a RepNCSPELAN4 module and multiple detection heads (specifically 3 detection heads) to form the detection output, where the RepNCSPELAN4 module is a key feature extraction-fusion module, which integrates the CSPNet Block module of YOLOv5, the Rep module of YOLOv6 and the ELAN module of YOLOv7, so that YOLOv9 can improve the detection accuracy while maintaining efficient calculation. The 3 detection heads use feature layers of different resolutions as input features to improve Yolov9's detection ability for multi-scale feature targets. The auxiliary branch is used to use multi-level auxiliary information to deal with the problem that the deepening of the neural network will lead to information bottlenecks and the loss function cannot generate reliable gradients. The multi-level auxiliary information is used to deal with the error accumulation problem caused by deep supervision. During model training, both the main branch and the auxiliary branch participate in training, and only the main network participates in reasoning. The CBLinear module is used to first adjust the feature layer channel and then divide it into equal parts by channel; the CBFuse module represents the following processing: based on a scale as the standard, the input feature layer is first interpolated, uniformly adjusted to an appropriate size, and merged by channel.

[0043] like Figures 3~9 As shown, considering the real-time performance of the calculation, this embodiment sets an FCBL feature extraction module at the input end (corresponding to the initial feature module) and the output end (corresponding to the bottleneck layer) of the RepNCSPELAN4 module respectively to implement feature extraction by using frequency adaptive dilated convolution FADConv. Compared with the traditional convolution method, more fine-grained features of the target can be obtained by using frequency adaptive dilated convolution FADConv, which improves the fine-grained feature extraction capability of the model, so that a balance between real-time performance and accuracy of the calculation can be achieved, thereby improving the recognition capability of the teacher model for the teacher. At the same time, only using FADConv convolution in the key feature extraction module can also achieve a balance between recognition accuracy and real-time performance.

[0044] Specifically, Figure 3 As shown, in this embodiment, the RepNCSPELAN4 module specifically includes an FCBL feature extraction module, a RepNCSP module and a CBL module, wherein the RepNCSP module includes multiple RepNBottleneck modules and performs feature fusion through a CSP (Cross-Stage Partial) structure, and each RepNBottleneck module is connected to a CBL module through a connection layer and then output to the FCBL feature abstraction module, as shown in FIG. Figure 4The FCBL feature extraction module uses frequency adaptive dilated convolution FADConv instead of traditional convolution operation, which can enhance the fine-grained feature extraction of the target and improve the feature recognition ability of the model, as shown in Figure 5 As shown in the figure, BN (Batch Normalization) is used to solve the problems of gradient disappearance, gradient explosion and training instability in the training process of deep learning, and SiLU represents the SiLU activation function. The CBL module uses Conv convolution operation to extract features to achieve feature abstraction, such as Figure 6 The RepNBottleneck module is a standard bottleneck module composed of RepACconvN basic computing blocks, as shown in Figure 7 As shown. Figure 8 As shown, the RepACconvN basic computing block contains a 3×3 ACconv convolution + BN layer, a 1×1 convolution + BN layer and a BN layer, and the three are connected in parallel to add their results.

[0045] In a specific application embodiment, the structure of implementing ACconv convolution is as follows: Fig. 9 , 10 As shown, in the training phase, the 3×3-BN structure is replaced by a 3×3-BN layer, a 1×3-BN layer, and a 3×1-BN layer multi-branch structure to train the model, and their outputs are summed, as shown in Fig. 9 As shown in Figure 2, after training, during inference, each 1×3-BN layer and 3×1-BN layer asymmetric kernel is added to the skeleton of the 3×3-BN branch, that is, the cross part of the square kernel, as shown in Figure 2. Fig.10 As shown in the figure, the three parallel convolution models during training are converted back to the same structure as the original ordinary convolution. Through the above-mentioned re-parameterized convolution process, the ordinary convolution can enhance the feature recognition ability of the convolution backbone part without increasing the calculation and changing the original structure, thereby further improving the overall model's feature representation ability for the scene images of teachers and their behaviors.

[0046] Traditional convolution usually acts as a high-pass filter, and the features generated by it often show a higher proportion of high-frequency components. This tendency leads to the use of a smaller overall expansion rate to maintain a higher effective bandwidth, but it sacrifices the size of the receptive field. A smaller receptive field will reduce the range of contact between the target pixel and the surrounding pixels, making it difficult to effectively extract the fine-grained features of the teacher model and reduce the model's ability to recognize the teacher. This embodiment uses frequency adaptive dilated convolution FADConv in the YOLOv network instead of traditional convolution operations, and can dynamically adjust the expansion rate of each position of the feature map in the feature map space according to the local frequency components of each position of the feature map, so that the feature map as a whole can obtain a suitable expansion rate while maintaining a higher effective bandwidth, and ultimately the teacher position detection model can obtain more fine-grained features and improve the teacher model's ability to recognize the teacher. Different from the traditional dilated convolution, the frequency adaptive dilated convolution FADConv in this embodiment achieves a balance between the effective bandwidth of the local frequency of the feature map and the local receptive field through three strategies: frequency selection (FreqSelect), adaptive dilation rate (AdaDR) and adaptive kernel (AdaKern), so that the model's ability to represent the local features of the feature map can be improved at a lower computational cost.

[0047] like Fig.11 As shown, the frequency selection module (FreqSelect) of this embodiment obtains the corresponding frequency channel map by Fourier transforming the feature map, and obtains different frequency channel maps (for example, Fig.11 is decomposed into 4 (configurable) frequency channel maps as decomposition features); then the feature map is convolved to obtain the selection map of each frequency channel (for example Fig.11 In the convolution operation, the convolution operation selects the feature map corresponding to the selected map in the convolution operation, and multiplies the selected map with the corresponding frequency channel to spatially reweight each frequency channel. This can balance the high-frequency and low-frequency components in the features, so that the frequency-adaptive dilated convolution can effectively learn a larger receptive field, thereby increasing the corresponding receptive field size through frequency selection (FreqSelect).

[0048] In frequency adaptive dilated convolution FADConv, the dilation rate is dynamically adjusted in a spatially variable manner through the adaptive dilation rate (AdaDR) to achieve a balance between effective bandwidth and receptive field. Fig.11 As shown in the figure, the adaptive dilation rate (AdaDR) is achieved by using convolution (Conv) + Relu activation function on the weighted features to adaptively assign a suitable dilation rate to each feature pixel, that is, by using convolution and Relu activation function to determine the adaptive dilation rate for each feature pixel, so that a better trade-off between a large receptive field and effective bandwidth can be achieved.

[0049] Traditional convolution kernel learning can capture features across different frequency bands, which is crucial for understanding complex visual patterns. However, once training is completed, they become static. In order to further enhance the effective bandwidth, the frequency adaptive dilated convolution FADConv in this embodiment can capture more high-frequency components through the adaptive kernel (AdaKern), thereby increasing the effective bandwidth. In the frequency adaptive dilated convolution FADConv of this embodiment, for the formation of low-frequency convolution, the average pooling and median pooling are summed and averaged to replace the traditional average pooling, that is, (avg+med) / 2 (average pooling + median pooling summed and averaged) is used to replace the traditional average pooling to reduce the impact of noise on low-frequency convolution; for the formation of high- and low-frequency convolution attention parameters, the dynamic weights (kernel weights) are predicted in turn through the first convolution layer conv1, the Relu activation function, the MLP layer, and the sigmod function layer, that is, the dynamic weights of the channels are predicted by conv1+Relu+conv1+sigmod. Compared with the traditional use of conv1+Relu+conv1+sigmod modules to determine channel weights, it can further integrate local semantics with overall semantic information, improve the feature representation ability of attention parameters, and thus improve the generalization ability of the model.

[0050] Specifically, Fig.11 As shown, in order to realize the adaptive kernel (AdaKern), the frequency adaptive dilated convolution FADConv of this embodiment decomposes the convolution kernel parameters into low-frequency convolution kernel parameters (low-frequency components) and high-frequency convolution kernel parameters (high-frequency components), and forms an adaptive convolution kernel by the low-frequency convolution kernel and the high-frequency convolution kernel, and introduces dynamic weighting to adjust the frequency response. When the average pooling operation is performed in the low-frequency convolution, the average pooling and the median pooling are summed and the average is taken, that is, the low-frequency convolution is averaged by the average pooling and the median pooling to replace the original average pooling, so as to reduce the influence of the noise signal in the original convolution on the low-frequency component; the global pooling and conv1, Relu, MLP layers, sigmod operations are used to predict the weights of the channels to form high and low-frequency convolution attention parameters, that is, the dynamic weights of the channels are predicted by a simple and lightweight global pooling plus conv1+Relu+MLP+sigmod, so that the kernel weights will change with the input, which can further integrate the local semantics with the global semantics and improve the feature representation ability of the attention parameters. Dynamically adjusting the ratio of high-frequency convolution kernel weights to low-frequency convolution kernel weights according to the input context can also enable the network to focus on specific frequency bands and adapt to the complexity of visual patterns in features. The dynamic frequency adaptation method can enhance the network's ability to capture low-frequency context and high-frequency local details, thereby increasing the effective bandwidth, thereby improving performance in segmentation tasks that require extracting diverse features across different frequencies.

[0051] In order to further increase the contextual semantic information of the features, this embodiment also includes a feature fusion module for fusing feature maps in the teacher position detection model. The feature fusion module is used to further fuse the feature maps to enhance the semantic connection between each pixel in the feature map and its surrounding pixels, further improving the model's ability to represent teacher features. The feature fusion module includes an ASSP module and a CA module, such as Fig.12 As shown in the figure, the ASSP module includes multiple feature extraction branches. Each branch realizes semantic feature extraction of different scales by adopting convolution (Conv), global average pooling (Avgpool) and dilated convolution Dcon with different expansion rates. Semantic feature extraction of different scales is realized by combining conv1×1, average pooling and dilated convolution Dconv with different expansion rates and merging them by channel to strengthen the semantic information connection between the target pixel and the surrounding pixels.

[0052] like Fig.12 As shown in the figure, the CA module fuses the feature maps extracted by the ASSP module through the residual structure to further enhance the semantic connection between pixels and surrounding pixels, and decomposes the channel attention into two one-dimensional feature encoding processes to aggregate features along the two spatial directions of X and Y. Specifically, the output of the residual structure first uses pooling kernels of size (H, 1) and (1, W) to encode each channel along the horizontal coordinate and vertical coordinate respectively (X Avg pool, Y Avg pool), connects the outputs of XAvg pool and Y Avg pool by channel, and extracts features through Conv2D convolution, and further abstracts features through BatchNorm (batch normalization) and Non-Linear (non-linear feature extraction layer), and then separates them into components in the X and Y directions, and reweights them after convolution (Conv2d) and Sigmod operations. In this way, long-range dependencies can be captured along one spatial direction while precise location information can be preserved along another spatial direction. The resulting feature maps are then individually encoded into a pair of direction-aware and position-sensitive attention maps, which can be complementarily applied to the input feature maps to enhance the representation of the teacher.

[0053] This embodiment realizes feature fusion by adopting ASSP+CA module. Compared with the traditional SPPELAN module, it can reduce the loss of feature extraction information while strengthening the semantic connection between central pixels and surrounding pixels, and improve the fusion between semantic features of different scales, thereby further improving the fine-grained feature extraction capability of the model.

[0054] In this embodiment, during the training process, the loss function of the teacher position detection model is Including classification loss function And the position regression loss function , the calculation expression is: (1) in, Represents the weight coefficient.

[0055] Optionally, the classification loss function The specific calculation can be obtained according to the following expression: (2) (3) in, represents the class-weighted focal loss function for the i-th class person, is the class weight factor of the student or teacher, where , m represents the number of identity categories, is the number of positive samples of the i-th category of identity, is the predicted value of the i-th category person, is the true label value of the i-th category person, is the balance factor, is the weight factor.

[0056] Optionally, the regression loss function The specific calculation can be obtained according to the following expression: (4) in, represents the bounding box loss, represents the distribution focal loss, , Represents the weight coefficient.

[0057] Step S03. Determine the teacher position area in the video image data based on the detection results of the first stage, input the determined teacher position area and classroom video image data into a pre-trained teaching behavior recognition model to detect the teacher's teaching behavior, and obtain the teacher's behavior category as the detection result output of the second stage.

[0058] After the first stage of teacher position detection, the teacher's position is accurately obtained, and the second stage of detection is further carried out to identify the teacher's current teaching behavior and determine whether it is a teaching behavior, such as identifying whether it is a gesture, standing to teach, writing on the blackboard, pointing to the multimedia screen or blackboard, operating teaching equipment, etc. Teacher position detection and teaching behavior recognition are detected using dedicated models to ensure the recognition accuracy and robustness of each task, thereby ensuring the accuracy and reliability of teacher teaching behavior recognition.

[0059] Since teaching behavior recognition involves understanding the semantic information of the entire image, in order to further strengthen the semantic connection between pixels and improve the accuracy of teaching behavior recognition, and considering the global information relationship modeling capability of Transformer and the need to ensure a certain degree of real-time performance when running the model, this embodiment uses the Swin Transformer architecture as the feature extraction model of the teacher behavior recognition model. When the teaching behavior recognition model performs teaching behavior detection, the input image is subjected to preliminary feature abstraction and block division using a convolutional neural network, and the feature maps after block division are respectively subjected to feature extraction using the Swin Transformer architecture, and finally the detection result of the teaching behavior category is obtained, wherein the W-MSA module and the MLP module in the SW-MSA module of the Swin Transformer architecture are implemented using the KAN module, that is, the KAN layer is used to replace the original MLP layer to implement the KAN-based Transformer architecture, which can improve the model's feature extraction capability for the teacher's character actions. The KAN module is formed by combining spline curves and MLP. That is, KAN is essentially a combination of spline curves and MLP. Therefore, it can absorb the advantages of both and give full play to the advantages of spline curves in high accuracy and strong interpretability in low dimensions, and MLP in convenient dimensional expansion.

[0060] Specifically, the steps of the teaching behavior recognition model in this embodiment for performing teaching behavior recognition are as follows: Step S301. Image segmentation: A convolutional neural network based on frequency adaptive dilated convolution FADConv is used to perform preliminary feature abstraction on the teacher image to enhance feature representation capability, and the image is segmented to adapt to the input of the Swin Transformer architecture; Step S302. Feature extraction: The Swin Transformer architecture is used to perform multiple feature extractions on the divided feature maps. After multiple executions, a balance between feature extraction performance and computing performance can be achieved.

[0061] Specifically, Fig.13As shown, x2 represents two modules, and x6 represents 6 modules. After the input image passes through the FADConv layer, step 1 (linear embedding, Swin Transformer Block (repeated twice)) is executed for feature extraction, wherein linear embedding is to achieve feature abstraction by linearly transforming the channel data of each pixel, and then step 2 (including patch merging, Swin Transformer Block (repeated twice)) is executed for further feature enhancement, and then the extracted features are executed in step 3 (including patch merging, Swin Transformer Block (repeated 6 times)) for further feature extraction, and finally, the output of step 3 is executed in step 4 (including patch merging, Swin TransformerBlock (repeated 2 times)) for final feature extraction to obtain the final classification result. It can be understood that the number of executions of Swin Transformer Block in the above steps can be configured according to actual needs.

[0062] Step S303. Classification and identification: Identify the teacher's teaching behavior category based on the features extracted in step S302.

[0063] like Fig.14 As shown, the Swin Transformer architecture based on KAN in this embodiment includes a KAN module, a W-MSA module and a SW-MSA module. KAN is used to replace the MLP in the traditional Transformer architecture, so that it can have a stronger function fitting ability to enhance the model's ability to fit the teacher's behavior and solve the problem of limited function fitting ability of the traditional MLP (multi-layer perceptron).

[0064] Specifically, the representation structure of the KAN module can be expressed as follows: (5) in, , is the activation function , is a spline function, x represents the input feature, represents the KAN module output corresponding to the input feature x, L represents the number of network layers, " indicates the composite operation of the function. Represents the weight coefficient.

[0065] Considering that the teaching behavior recognition model is a classification model based on the Swim Transformer architecture, and teachers may perform multiple teaching behaviors at the same time, a multi-label classification loss function can be used in the training process of the teaching behavior recognition model. To optimize the teaching behavior recognition model, the calculation formula can adopt the same principle as the above formula (2).

[0066] Embodiment 2: This embodiment is basically the same as Embodiment 1, except that it also includes a classroom teaching quality evaluation step: calculating a first indicator based on the first stage detection results and calculating a second indicator based on the second stage results, and evaluating the teacher's teaching behavior based on the first indicator and the second indicator, the first indicator includes the number of times the teacher is detected, and the second indicator includes the total number of times the teacher's teaching behavior is detected. That is, the steps of the classroom teacher teaching behavior detection method of this embodiment include: Step S01. Receive classroom video image data collected by the camera in real time; Step S02. Input the received classroom video image data into a pre-trained teacher position detection model to detect the teacher's position as the first-stage detection result; the teacher position detection model is a target detection model constructed based on the YOLOv network, and some convolution operations in the YOLOv network adopt frequency adaptive dilated convolution FADConv, the frequency adaptive dilated convolution FADConv decomposes the feature map into multiple frequency channels, obtains a selection map of each frequency channel through a convolution operation, multiplies the selection map with the corresponding frequency channel to obtain a weighted feature to achieve frequency selection, and performs a convolution operation on the weighted feature using an adaptive dilation rate and an adaptive convolution kernel to obtain an output feature layer; Step S03. Determine the teacher's position area in the video image data based on the detection results of the first stage, input the determined teacher's position area and the classroom video image data into a pre-trained teaching behavior recognition model to detect the teacher's teaching behavior, and obtain the teacher's behavior category as the detection result output of the second stage.

[0067] Step S04. Calculate the first indicator based on the detection results of the first stage and calculate the second indicator based on the results of the second stage, and evaluate the teacher's teaching behavior based on the first indicator and the second indicator. The first indicator includes the number of times the teacher is detected, and the second indicator includes the total number of times the teacher's teaching behavior is detected.

[0068] This embodiment first adopts a two-stage detection method to perform teacher position detection and teaching behavior recognition in sequence. Each task is detected using a dedicated model to ensure the recognition accuracy and robustness of each task, thereby ensuring the accuracy of the teacher position detection task and its teaching behavior recognition task, and achieving accurate positioning of the teacher's identity in the classroom and accurate and robust recognition of its related teaching behaviors. The two-stage detection results are then combined to achieve accurate auxiliary evaluation of the teacher's teaching behavior, which can be beneficial to improving teaching quality.

[0069] In order to effectively evaluate the teaching behavior of teachers and conduct quantitative evaluation of teaching behavior, this embodiment evaluates their teaching behavior from two aspects: effective teaching time and teaching initiative. By counting the effective teaching time of teachers in class, some lateness and early departure behaviors can be identified and teachers' teaching behavior can be standardized. Teaching initiative can reflect the attitude of teachers towards teaching tasks.

[0070] Optionally, the effective teaching time of teachers in class can be evaluated based on the number of times the teacher is detected , the teaching initiative can be evaluated based on the total category of teacher teaching behaviors detected , for example, the following calculation formulas can be used: (6) (7) in, Indicates the total number of detections. represents the total number of times the teacher is detected, m represents the total types of the teacher’s teaching behaviors, Represents the total number of times each teaching behavior is detected.

[0071] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the methods of the above-mentioned embodiments 1 and 2.

[0072] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0073] This embodiment further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method of the above-mentioned embodiments 1 and 2 is implemented.

[0074] Those skilled in the art should understand that the above-mentioned embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A two-stage classroom teacher teaching behavior detection method, characterized in that the steps include: Receive classroom video image data collected by the camera in real time; The received classroom video image data is input into the pre-trained teacher position detection model to detect the teacher's position as the first stage detection result; The teacher position detection model is a target detection model built based on the YOLOv network, and some convolution operations in the YOLOv network adopt frequency adaptive dilated convolution FADConv, which performs Fourier transform on the feature map to obtain the corresponding frequency channel map, and decomposes it into multiple frequency channel maps according to the frequency, obtains the selection map of each frequency channel by convolution operation on the feature map, multiplies the selection map with the corresponding frequency channel map to obtain weighted features to achieve frequency selection, and performs convolution operation on the weighted features using adaptive dilation rate and adaptive convolution kernel to obtain output feature layer; Based on the detection results of the first stage, the teacher's position area in the video image data is determined, and the determined teacher's position area and the classroom video image data are input into a pre-trained teaching behavior recognition model to perform teacher's teaching behavior detection, and the teacher's behavior category is obtained as the detection result output of the second stage.

2. The double-stage classroom teacher teaching behavior detection method according to claim 1 is characterized in that: The adaptive expansion rate is determined for each feature pixel by using convolution and Relu activation function, the convolution kernel parameters are decomposed into low-frequency convolution kernel parameters and high-frequency convolution kernel parameters, and the ratio of the high-frequency convolution kernel weight to the low-frequency convolution kernel weight is adjusted according to the input context to form the adaptive convolution kernel. When the average pooling operation is performed in the low-frequency convolution, the average is taken by summing the average pooling and the median pooling, and the channel weight is predicted in sequence through the first convolution layer conv1, the Relu activation function, the MLP layer, and the sigmoid function layer.

3. The double-stage classroom teacher teaching behavior detection method according to claim 1 is characterized in that: The teacher position detection model is constructed based on the YOLOV9 network. The backbone network in the YOLOV9 network includes a RepNCSPELAN4 module and multiple detection heads. The bottleneck layer in the RepNCSPELAN4 module and the convolution operation in the YOLOV9 initial feature module adopt the frequency adaptive dilated convolution FADConv.

4. The double-stage classroom teacher teaching behavior detection method according to claim 1 is characterized in that: The teacher position detection model also includes a feature fusion module for fusing feature maps, and the feature fusion module includes an ASSP module and a CA module. The ASSP module realizes semantic feature extraction of different scales by adopting convolution, average pooling and dilated convolution with different expansion rates. The CA module fuses the feature maps extracted by the ASSP module through a residual module, and decomposes the channel attention into two one-dimensional feature encoding processes to aggregate features along two spatial directions respectively.

5. The double-stage classroom teacher teaching behavior detection method according to claim 1 is characterized in that: The teaching behavior recognition model is based on the Swin Transformer architecture as a feature extraction model. When the teaching behavior recognition model performs teaching behavior detection, the input image is preliminarily abstracted and divided into blocks using a convolutional neural network, and the feature maps after block division are respectively subjected to feature extraction using the Swin Transformer architecture, and finally the detection result of the teaching behavior category is obtained. The W-MSA module of the Swin Transformer architecture and the MLP module in the SW-MSA module are implemented using the KAN module, and the KAN module is formed by combining a spline curve and an MLP.

6. The double-stage classroom teacher teaching behavior detection method according to claim 5 is characterized in that: The representation structure of the KAN module is as follows: in, , is the activation function , is a spline function, x represents the input feature, represents the KAN module output corresponding to the input feature x, L represents the number of network layers, represents the weight coefficient, Represents a composite operation of a function.

7. The method for detecting the teaching behavior of teachers in a two-stage classroom according to any one of claims 1 to 6, characterized in that: The loss function of the teacher position detection model is Including classification loss function And the position regression loss function , the calculation expression is: in, represents the weight coefficient, represents the class-weighted focal loss function for the i-th class person, is the class weight factor of the student or teacher, where , represents the number of identity categories, is the number of positive samples of the i-th category of identity, is the predicted value of the i-th category person, is the true label value of the i-th category person, is the balance factor, is the weight factor, represents the bounding box loss, represents the distribution focal loss, , Represents the weight coefficient.

8. The method for detecting the teaching behavior of teachers in a two-stage classroom according to any one of claims 1 to 6, characterized in that: It also includes calculating a first indicator based on the first stage detection results and calculating a second indicator based on the second stage results, and evaluating the teacher's teaching behavior based on the first indicator and the second indicator. The first indicator includes the number of times the teacher is detected, and the second indicator includes the total number of times the teacher's teaching behavior is detected.

9. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images