A classroom behavior recognition method based on an improved YOLOv12 model

Abstract

The application discloses a classroom behavior recognition method based on an improved YOLOv12 model, which proposes an improved YOLOv12 model according to a classroom behavior recognition application scene, designs an A2C2f_FRFN module combined with an A2C2f and a feature refinement feedforward network FRFN module at a backbone network and a neck network, and through multi-stage linear transformation, convolution operation and a gating mechanism, can strengthen the attention degree to slight action or posture change and improve the ability to capture details in classroom behavior recognition. The C3k2_SAVSS module combined with the C3k2 and the structure perception visual state space module SAVSS is designed at the backbone network and the neck network, and through direction scanning (horizontal, vertical and diagonal line) and a structured state space model, long-range dependence and global features can be captured, and the combination of the two can comprehensively integrate local details and global context.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and the field of education, and relates to a classroom behavior recognition method, in particular to a classroom behavior recognition method based on an improved YOLOv12 model. BACKGROUND

[0002] In the field of modern education, with the in-depth development of educational informatization, the demand for fine analysis and management of classroom teaching process is increasingly urgent. Classroom behavior recognition, as a key technology to achieve this goal, has extremely important significance. It can objectively and comprehensively obtain students' behavior performance information in the classroom, providing strong support for teachers to adjust teaching strategies, optimize teaching methods, and for education researchers to deeply understand the teaching process.

[0003] Traditional classroom behavior recognition methods have many limitations. For example, relying on manual observation and recording of students' classroom behavior not only consumes a lot of manpower and time, but also is highly subjective, prone to omissions and biases, and difficult to fully and accurately reflect students' real classroom performance. With the rise of computer vision technology, some classroom behavior recognition methods based on traditional image processing and machine learning have begun to be applied, but these methods perform poorly in feature extraction and model generalization ability, and face complex and variable classroom scenes such as different lighting conditions, diverse student postures, and frequent occlusion, with low recognition accuracy, failing to meet the needs of practical applications.

[0004] In recent years, deep learning technology has made significant breakthroughs in target detection and image recognition, bringing new opportunities for classroom behavior recognition. Target detection algorithms based on deep learning are mainly divided into one-stage target detection algorithms and two-stage target detection algorithms. Although two-stage algorithms have high accuracy, they are slow in detection speed and have high computational complexity; one-stage algorithms only need to complete the detection process once, with smaller computational load, making them more suitable for real-time application scenarios such as classroom behavior recognition, which requires real-time analysis of video streams. In one-stage algorithms, the YOLO series of algorithms have achieved a good balance between accuracy and speed, and have received widespread attention and application.

[0005] However, existing YOLO series algorithms still have some problems when dealing with classroom behavior recognition tasks. Classroom scenes have characteristics such as high personnel density, diverse behaviors, frequent occlusions, and large scale variations, and existing YOLO models lack the ability to capture global information, and their detection accuracy and robustness need to be improved when facing occlusion and multi-scale variation scenarios. For example, when multiple students' body parts are mutually occluded, the model is prone to miss detection or false detection; for students with different distances from the camera, due to large scale differences, the model performs poorly in detecting small-scale targets. SUMMARY

[0006] In order to solve the above problems existing in the processing of the classroom behavior recognition task of the existing YOLO series algorithm, the application provides a classroom behavior recognition method based on an improved YOLOv12 model.

[0007] The purpose of the application is realized by the following technical solutions:

[0008] A classroom behavior recognition method based on an improved YOLOv12 model comprises the following steps:

[0009] Step one, obtaining a data set of classroom behavior images;

[0010] Step two, dividing the data set into a verification set, a training set and a test set, and processing into a network model recognition format of YOLOv12;

[0011] Step three, training the data set using a classroom behavior recognition detection model of the improved YOLOv12 model, wherein the improved YOLOv12 model increases a P2 detection head on a detection layer, and an A2C2f_FRFN module combining A2C2f and a feature refinement feedforward network FRFN module is designed and used at a backbone network and a neck network, and a C3k2_SAVSS module combining C3k2 and a structure-aware visual state space SAVSS module is designed and used at the backbone network and the neck network;

[0012] Step four, training the classroom behavior recognition detection model using the training set and the verification set, and evaluating the performance of the classroom behavior recognition detection model using the test set after obtaining the optimal detection model;

[0013] The training set and the verification set are input into the classroom behavior recognition detection model of the improved YOLOv12 model, and the number of training times is set, with the increase of the number of training times, the loss function curve of the detection model gradually converges, when the loss function curve converges and stabilizes, the detection model is trained to the optimal state, and the optimal model weight file is saved, the test set is input into the trained optimal behavior recognition detection model, and the detected image is output, wherein the detected image includes the type of each detection target, and the position of each target in the target detection image is marked;

[0014] Step five, combining monitoring and the optimal detection model to perform real-time recognition and detection on the classroom behavior of students.

[0015] Compared with the prior art, the application has the following advantages:

[0016] The present invention proposes an improved YOLOv12 model based on the application scenario of classroom behavior recognition. The A2C2f_FRFN module that combines A2C2f with the feature refinement feedforward network FRFN module is innovatively designed and used in the backbone network and neck network. Through multi-stage linear transformation, convolution operation and gating mechanism, it can enhance the attention to small movements or posture changes and improve the ability to capture details in classroom behavior recognition. The C3k2_SAVSS module that combines C3k2 with the structure perception visual state space module SAVSS is innovatively designed and used in the backbone network and neck network. Through directional scanning (horizontal, vertical, diagonal) and structured state space model, it can capture long-range dependencies and global features. The combination of the two can fully integrate local details and global context. For example, in the classroom, it can accurately extract local action details such as students writing and raising their hands, and can also associate previous and subsequent behaviors to enhance the richness of feature expression. The multi-directional scanning strategy of SAVSS enables the module to capture multi-angle behaviors in the classroom more comprehensively, and can generate scenes that adapt to different lighting and occlusion levels through dynamic parameters. For example, when a student raises his hand sideways, multi-directional feature fusion can still accurately extract action features. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of the classroom behavior recognition method based on the improved YOLOv12 model;

[0018] Figure 2 This is a detection model diagram of the classroom behavior recognition method based on the improved YOLOv12 model;

[0019] Figure 3 This is a schematic diagram of the A2C2f_FRFN module structure;

[0020] Figure 4 This is a schematic diagram of the ABlock_FRFN module structure;

[0021] Figure 5 This is a schematic diagram of the C3k2_SAVSS module structure;

[0022] Figure 6 This is a schematic diagram of the SAVSS module structure;

[0023] Figure 7 It is a schematic diagram of the GBC module structure;

[0024] Figure 8 For the FRFN2D module;

[0025] Figure 9 Schematic diagram of the PAF module structure. DETAILED DESCRIPTION

[0026] The technical solutions of the present application are further described below with reference to the drawings, but are not limited thereto, and any modification or equivalent replacement of the technical solutions of the present application without departing from the spirit and scope of the technical solutions of the present application shall be covered in the protection scope of the present application.

[0027] A classroom behavior recognition method based on an improved YOLOv12 model, the method comprising the following steps:

[0028] Step one, obtaining a data set of classroom behavior images.

[0029] Step two, dividing the data set into a verification set, a training set, and a test set, and processing into a network model recognition format of YOLOv12.

[0030] Step three, training the data set using a classroom behavior recognition detection model of the improved YOLOv12 model, the improved YOLOv12 model adding a P2 detection head on the detection layer, and designing and using an A2C2f_FRFN module combining A2C2f and a feature refinement feedforward network FRFN module at the backbone network and the neck network, and designing and using a C3k2_SAVSS module combining C3k2 and a structure perception visual state space module SAVSS at the backbone network and the neck network.

[0031] In this step, the implementation process of the A2C2f_FRFN module is as follows: the input feature vector X is first imported into the initial convolutional layer constructed based on the parent class initialization logic, which implements channel compression on the input feature map through low-rank mapping, performs multiplication and accumulation operation of weights and corresponding position elements by means of sliding convolution of convolution kernel in feature map space domain, completes dimension transformation from input channel to hidden channel, and extracts feature information with basic representation ability. The feature map processed by this convolution is stored in a specific list structure. Then enter the loop processing flow composed of ABlock_FRFN modules, and sequentially input the elements at the end of the list into each module sequence. The feature vector X2 generated by the ABlock_FRFN module processing is residual mapped and fused with the original input feature vector X, and finally the fused feature Y is output. Through the above module architecture design and data processing flow, efficient extraction, screening and fusion of complex input features are realized, the discriminability and robustness of feature expression are significantly enhanced, more discriminative and stable feature representation is provided for high-level semantic tasks such as classroom behavior recognition, and the capture and processing ability of the model to subtle feature changes in complex scenes is effectively improved, thereby guaranteeing the accuracy and reliability of the recognition task from the feature level. The forward propagation mathematical expression is:

[0032] Y=X+X2

[0033] The implementation process of the ABlock FRFN module is as follows: the upper-level input feature vector X is first processed by the regional attention module inherited from the ABlock class. The regional attention mechanism assigns weights to the features, highlighting key information, and obtains the feature vector Y mlp1 , which is first connected with the original upper-level input feature vector X to obtain X1. This operation aims to alleviate the gradient vanishing problem and enhance the model's ability to capture key features. X1 is transmitted to the multi-layer perceptron module (a sequence composed of two convolutional layers) in the parent class ABlock for feature transformation, obtaining the feature vector Y mlp1 , which is secondly connected with X1 to obtain X2. X2 enters the FRFN2D module for processing, and finally obtains the feature vector and is connected with X2 in residual connection to obtain the output Output. Through the series of steps of initial convolution dimension reduction, regional attention mechanism, multi-layer perceptron transformation, and feature refinement of the FRFN2D module, the ABlock FRFN module realizes multi-stage fine processing and deep fusion of the input features, significantly enhancing the expression ability of the features and providing more valuable feature representation for subsequent tasks. The forward propagation mathematical expression is:

[0034] X1=X+Y att1

[0035] X2=X1+Y mlp 1

[0036] Output=X2+FRFN2D

[0037] The FRFN2D module implementation process is as follows: the input feature vector X of the previous stage is first divided into two parts X1 and X2 along the channel dimension, 3*3 convolution is performed on X1, which extracts the local detail information of the feature through local convolution operation, enhances the spatial correlation and local representation ability of the feature, and then is spliced with X2 along the channel dimension, which introduces local detailed features while retaining part of the original feature integrity, maintains the overall structure and information diversity of the feature. The spliced feature is expanded by linear1 (including 1*1 convolution and activation function) to expand the channel number, thereby widening the feature expression space by increasing the channel dimension, so that the model can capture more rich feature patterns and semantic information, and the activation function further introduces non-linear characteristics to improve the fitting ability of the model to complex feature relationships. Subsequently, the feature is divided into two parts X3 and X4, and depth separable convolution is performed on X3, which extracts the local spatial information and inter-channel correlation of the feature while reducing the amount of calculation, and then is multiplied with X4 element by element, which selects the more critical feature information for subsequent tasks through the gating mechanism, suppresses redundant or irrelevant features, enhances the discriminability and effectiveness of the feature. Finally, linear2 (1*1 convolution) is used to restore the channel number to the input dimension dim, so that the feature dimension adapts to the requirements of subsequent tasks, integrates and compresses the feature without losing key information, and obtains the final output Output, which provides refined and high-expression feature representation for subsequent processing of the model. The forward propagation mathematical expression is:

[0038] X1=partialconv(X1)

[0039] X=Concat((X1,X2),dim=1)

[0040] X=linear1(X)

[0041] X3,X4=X·chunk(2,dim=1)

[0042] X3=dwconv(X3)

[0043] X=X3X4

[0044] Output=linear2(X)

[0045] In this step, the implementation process of the C3k2_SAVSS module is as follows: the input feature X of the previous stage is adjusted by an initial convolution layer to generate hidden features X h =Conv(X), where the hidden channel number is c, and X h is input into a sequence composed of n SAVSS modules, each SAVSS performs a specific transformation on the feature and outputs an intermediate feature F i (X h ); finally, the final output Fn (X h ) and the original upper-level input feature X to obtain the fusion result Y. The forward propagation mathematical expression is:

[0046] X h = Conv(X)

[0047]

[0048] Y = X + F n (X h )

[0049] The SAVSS module implementation process is as follows: first, the upper-level input feature X is subjected to two GBC (Grouped Bottleneck Convolution) operations. The module significantly reduces the computational complexity through the grouped depth separable convolution structure, and enhances the local feature expression ability through the multi-branch bottleneck design. It contains four parallel sub-modules: two 3x3 depth convolution branches for capturing local spatial details, and two 1x1 convolution branches for cross-channel information interaction, which realize efficient feature enhancement with group normalization (GN) and ReLU activation function, and finally retain the original information through residual connection to avoid gradient disappearance. The enhanced feature map X gbc is unfolded from the spatial dimension into a sequence form and sent to the SAVSS_2D module to realize long-range dependence modeling through the structured state space model (SSM) combined with the multi-direction scanning strategy: horizontal scanning captures the correlation between pixels in the row, vertical scanning establishes the dependence within the column, and double diagonal scanning extracts global diagonal features. The outputs of the four scanning paths are dynamically weighted and fused to ensure that the model can perceive both local structure and global context; during the scanning process, the transition matrix A generated dynamically and the content-aware projection weight B are adjusted adaptively according to the input feature, so that the state transition process has input sensitivity, significantly improving the modeling ability of complex spatial patterns. The serialized feature Y ssm output by the SSM gbc is deeply fused through the PAF (Position-Aware Fusion) module. The module first generates a key-value pair of base features and guide features through double-feature transformation, calculates a spatial similarity graph S to quantify the position correlation, and then performs soft assignment fusion of the original features and SSM features with S as the weight, which not only retains the bottom-level details but also injects global structured information, especially suitable for feature complementation in occlusion or small target scenarios. In the final stage, linear transformation and residual connection are used to stabilize the training process: group normalization (GN) alleviates the internal covariate shift, linear layers improve the feature dimension adaptability, and skip connection ensures lossless information transmission to output Y outThe entire module is synergized by the local enhancement of GBC, the multi-directional global modeling of SAVSS_2D and the dynamic fusion of PAF, which greatly improves the mIoU index of dense prediction task compared with the traditional convolution module while reducing the amount of calculation. The forward propagation mathematical expression is:

[0050] X gbc = GBC(GBC(X))

[0051]

[0052] S = σ(Adapterr channel (Conv(X gbc )) ⊙ Adapter channel (Conv(Reshape(Y ssm ))))

[0053] Y paf = (1-S) ⊙ X gbc +S ⊙ Reshape(Y ssm )

[0054] Y out = Reshape(Linear(GN(Y paf )))+X gbc

[0055] Where σ is the Sigmoid function, and ⊙ represents element-wise multiplication.

[0056] The implementation process of the GBC (Grouped Bottleneck Convolution) module is as follows: first, the original upper-level input feature X is subjected to GBC operation. This module can significantly reduce the computational complexity by means of grouped deep separable convolution structure. Its internal adopts a multi-branch bottleneck design, which includes four parallel sub-modules: two 3x3 deep convolution branches, mainly used to capture local spatial details and mine subtle spatial structure information in the feature map. Two 1x1 convolution branches are used for cross-channel information interaction to realize feature fusion and transformation between channels. These sub-modules cooperate with group normalization (GN) operation to alleviate the internal covariate shift problem, and then introduce nonlinearity through ReLU activation function, thereby realizing efficient feature enhancement. Finally, the original input information and the processed feature information are added through residual connection to obtain the output, which can not only preserve the original information and effectively avoid the gradient vanishing problem, but also help the model to learn more rich feature differences. Where X gbc is the enhanced feature map after two GBC operations, and the first GBC operation obtains X gbc1 . The forward propagation mathematical expression is:

[0057] Xgbc1 =GBC(X)

[0058] X gbc =GBC(X gbc1 )

[0059] Y=X gbc1 +X

[0060] The implementation process of the SAVSS_2D module is as follows: First, the enhanced feature map X processed by the GBC module is gbc It is expanded into a sequence form from the spatial dimension as the input of the SAVSS_2D module. The SAVSS_2D module uses the structured state space model (SSM) combined with a multi-directional scanning strategy to process the input features. The multi-directional scanning strategy includes horizontal scanning, vertical scanning, and dual diagonal scanning. The horizontal scan processes the feature sequence row by row to capture the long-range dependencies between pixels in the row direction of the feature map; the vertical scan processes the feature sequence column by column to mine the long-range dependencies in the column direction; the dual diagonal scan scans the feature map from two diagonal directions to obtain the correlation information between pixels in the diagonal direction. During the scanning process, the module dynamically generates the transfer matrix A based on the input features. d and projection weight B d (d represents different scanning directions.) These parameters are adaptively adjusted according to the input features, so that the state transition process can better adapt to the changes in features, thereby improving the modeling ability of complex spatial patterns. For each scanning direction d, there is a corresponding structured state space model SSM d Based on the input feature sequence, transfer matrix and projection weight, the calculation is performed through the state transition rule. Finally, the outputs of the four scanning paths are dynamically weighted and fused. The weight α is calculated for the output of each scanning direction. d , and satisfy the condition that the weight sum is 1. Through weighted summation, the final serialized feature Y is obtained ssm . Its forward propagation mathematical expression is:

[0061]

[0062] z t,d =A d z t-1,d +B d x t,d

[0063] Y d =SSM d (Norm(Reshape(X gbc )),A d ,B d )

[0064]

[0065] The implementation process of the PAF module is as follows: The PAF module uses the serialized feature Y output by the SAVSS_2D module ssm And the original spatial feature X after processing by the GBC module gbc As input, through two-way feature transformation, X gbc Convert to base feature, Y ssm Convert it into a guide feature and generate a key-value pair. Based on this, the spatial similarity graph S is calculated. This process can quantify the correlation of features in spatial positions. Then, the original feature X is weighted with S. gbc and SSM feature Y ssm Perform soft assignment fusion and finally obtain the fused feature Y paf . It can retain X gbc The underlying detailed information can be injected into Y ssm The global structured information of is complementary to the features of small target behaviors and targets in classroom images. Its forward propagation mathematical expression is:

[0066] Y paf =(1-S)⊙X gbc +S⊙Reshape(Y ssm )

[0067] Step 4: Use the training set and validation set to train the classroom behavior recognition detection model, and use the test set to evaluate the performance of the classroom behavior recognition detection model to obtain the optimal detection model.

[0068] Step 5: Combine monitoring with the optimal detection model to conduct real-time identification and detection of students' classroom behavior. The specific steps are as follows:

[0069] Step 5: Place high-definition surveillance cameras in the classroom environment to capture key information about students' movements, postures, and facial expressions from different angles.

[0070] Step 52: Transmit the video stream data collected in real time by the surveillance camera to the computer processing terminal;

[0071] Step 53: After receiving the video stream data, the computer terminal uses the optimal detection model obtained in step 4 to extract and analyze the features of each input frame. Once the optimal classroom behavior detection model identifies that a student has typical classroom behavior, it will provide real-time feedback in a visual manner on the computer display interface.

[0072] Step 54: Use differentiated labeling methods for different behaviors.

[0073] Example:

[0074] The embodiment provides a classroom behavior recognition method based on an improved YOLOv12 model, as shown in the formula (I). Figure 1 The method comprises the following steps:

[0075] Step one, obtain the data set of the classroom behavior image.

[0076] The student classroom behavior data set (SCB-Dataset) provides a real and accurate student behavior reflection, contains 184,000 labels and 42,000 images, covers three behaviors: raising hands, reading and writing. A part is selected for use.

[0077] Step two, use a computer to preprocess the data set. The data set is divided into a verification set, a training set and a test set, and finally processed into a YOLOv12 network model recognition format.

[0078] The hand data set is divided into a training set, a verification set and a test set according to a set proportion 8:1:1 by using a python code, wherein the training set is used to train the model, the verification set is used for evaluation in the training process, the test set is used to evaluate the performance of the model, and the training set, the verification set and the test set are processed into a YOLOv12 network model recognition format.

[0079] Step three, use the improved YOLOv12 model of the classroom behavior recognition detection model to train the data set, the improved YOLOv12 adds a P2 detection head on the detection layer, and an A2C2f_FRFN module combining A2C2f and a feature refinement feedforward network FRFN module is innovatively designed and used at the backbone network and the neck network, and a C3k2_SAVSS module combining C3k2 and a structure perception visual state space module SAVSS is innovatively designed and used at the backbone network and the neck network. As shown in the formula (II). Figure 2

[0080] A P2 detection head is added on the detection layer, so that the hand image is transmitted into the Backbone network to extract four different feature maps, and four sizes 120x120, 64x64, 32x32 and 16x16 are respectively used for detecting four different sizes of targets, i.e. micro, small, medium and large.

[0081] ​An A2C2f_FRFN module is designed at the backbone network and the neck network, which combines A2C2f with a feature refinement feedforward network (FRFN) module. The spatial feature extraction capability of the model is enhanced by introducing a strip attention mechanism. Through multi-stage linear transformation, convolution operation and gating mechanism, the attention to minor actions or posture changes and the ability to capture details can be enhanced in classroom behavior recognition. The input tensor is first introduced into the initial convolution layer constructed based on the parent class initialization logic. This layer performs channel compression on the input feature map through low-rank mapping. By means of sliding convolution of the convolution kernel in the feature map space domain, the product accumulation operation of the weight and the corresponding position element is performed, the dimension transformation from the input channel to the hidden channel is completed, and the feature information with basic representation ability is extracted. The feature map processed by this convolution is stored in a specific list structure. Then it enters the loop processing procedure composed of ABlock_FRFN modules. The elements at the end of the list are sequentially transmitted into each module sequence as input. The feature vector generated by the ABlock_FRFN module processing is residual mapped and fused with the original input feature vector, and finally the fused feature is output. Through the above module architecture design and data processing procedure, efficient extraction, screening and fusion of complex input features are realized, the discriminability and robustness of feature expression are significantly enhanced, more discriminative and stable feature representation is provided for high-level semantic tasks such as classroom behavior recognition, and the ability to capture and process subtle feature changes in complex scenarios is effectively improved. The accuracy and reliability of the recognition task are guaranteed from the feature level. As shown in Figure 3

[0082] ​The ABlock FRFN module helps the model focus on the most relevant parts of the input data. In the process of behavior recognition, some moments or actions contain more information than others. Attention mechanisms can allow the model to adaptively assign more weight to key frames or key actions. The input features are first processed by the regional attention module inherited from the ABlock class. The regional attention mechanism assigns weights to the features, highlighting key information, resulting in a "regional attention feature vector". This is the first residual connection with the original input features, resulting in a "first residual connection feature vector". The purpose of this operation is to alleviate the problem of gradient disappearance, while enhancing the model's ability to capture key features. The "first residual connection feature vector" is passed into the multi-layer perceptron module in the parent class ABlock for feature transformation, resulting in a "multi-layer perceptron feature vector". This is the second residual connection with the "first residual connection feature vector", resulting in a "second residual connection feature vector". The "second residual connection feature vector" is processed by the FRFN2D module, resulting in a feature vector that is then residual connected with the "second residual connection feature vector" to produce the output. Through the series of steps of initial convolution dimension reduction, regional attention mechanism, multi-layer perceptron transformation, and FRFN2D module feature refinement, the ABlock FRFN module achieves multi-stage fine processing and deep fusion of input features, significantly enhancing the expressive power of features and providing more valuable feature representations for subsequent tasks. As shown in Figure 4

[0083] ​The FRFN2D module enhances features by means of Partial Conv and Gate Mechanism, achieving higher fine-grained behavior detection accuracy and real-time processing capability. When processing input features, it first divides them into two parts along the channel dimension. One part is subjected to a 3x3 convolution operation, which is a local convolution that can extract local detail information of features, enhance the relevance of features in space and local representation capability. Then, the convoluted part of the features is spliced with the other part along the channel dimension. This not only preserves the integrity of some original features, but also introduces local detailed features, maintains the overall structure of the features and the diversity of information. The spliced features enter the linear1 module. This module contains a 1x1 convolution and an activation function. The 1x1 convolution is used to expand the channel number, widen the spatial range of feature expression, and enable the model to capture more rich feature patterns and semantic information. The activation function further introduces nonlinearity, improving the model's fitting ability for complex feature relationships. Then, the processed features are evenly divided into two parts. One part is subjected to depth separable convolution, which can extract local spatial information of features and the correlation between channels while reducing computational complexity. Subsequently, the convoluted part of the features is multiplied with the other part element by element. This gate mechanism can filter out key feature information for subsequent tasks, suppress redundant or irrelevant features, and enhance the discriminability and effectiveness of features. Finally, the linear2 module (containing a 1x1 convolution) is used to restore the channel number to the input dimension, so that the feature dimension can adapt to the needs of subsequent tasks. Without losing key information, the features are integrated and compressed to obtain the final output result. This output result provides refined and high-expression feature representation for subsequent processing of the model. As shown in Figure 8 .

[0084] At the backbone network and neck network, the C3k2_SAVSS module combining C3k2 and structure-aware visual state space module SAVSS is designed and used. This module can capture long-range dependencies and global features through directional scanning (horizontal, vertical, diagonal) and structured state space model. The combination of the two can fully integrate local details and global context. For example, in a classroom, it can accurately extract local action details such as student writing and hand raising, and also correlate previous and subsequent behaviors to enhance the richness of feature expression. First, the input feature map is adjusted by the initial convolution layer to generate hidden features. Then, the generated hidden features are input into a sequence composed of n SAVSS modules. Each SAVSS module performs a specific transformation on the features and outputs intermediate features. Finally, the final output features are added to the original input features to obtain the fusion result. As shown in Figure 5 .

[0085] The SAVSS module greatly improves the mIoU index of dense prediction tasks compared to traditional convolution modules by reducing the amount of calculation through the synergistic effect of three techniques: local enhancement of GBC, multi-directional global modeling of SAVSS_2D, and dynamic fusion of PAF. First, the input feature map is subjected to two GBC (Grouped Bottleneck Convolution) operations. This module significantly reduces computational complexity through a grouped deep separable convolution structure, while enhancing local feature expression capability using a multi-branch bottleneck design. It contains four parallel sub-modules: two 3x3 deep convolution branches to capture local spatial details, and two 1x1 convolution branches for cross-channel information interaction, which work together with group normalization (GN) and ReLU activation functions to achieve efficient feature enhancement. Finally, the enhanced feature map is obtained through residual connection to preserve the original information and avoid gradient disappearance. The enhanced feature map is then unfolded from the spatial dimension to a sequence form and fed into the SAVSS_2D module, which models long-range dependencies through a structured state space model (SSM) combined with a multi-directional scanning strategy. Horizontal scanning captures intra-row pixel correlations, vertical scanning establishes intra-column dependencies, and double diagonal line scanning extracts global diagonal line features. The outputs of the four scanning paths are dynamically weighted and fused to ensure that the model can perceive both local structures and global contexts. During the scanning process, the transition matrix and content-aware projection weights are self-adaptively adjusted according to the input features, making the state transition process input-sensitive and significantly improving the modeling capability of complex spatial patterns. The serialized features output by the SSM are deeply fused with the original spatial features through the PAF (Position-Aware Fusion) module. This module first generates key-value pairs of base features and guide features through double-feature transformation, calculates a spatial similarity map to quantify position correlation, and then performs soft assignment fusion of the original features and SSM features using the spatial similarity map as the weight, preserving the underlying details while injecting global structured information, especially suitable for feature complementation in occlusion or small target scenarios. In the final stage, linear transformation and residual connection are used to stabilize the training process: group normalization (GN) alleviates internal covariate shift, linear layers increase feature dimension adaptability, and skip connection ensures lossless information transmission before outputting the final result. As shown in Figure 6

[0086] ​The GBC module enhances the expression ability of local details through a multi-branch lightweight bottleneck convolution structure. With the help of grouped depthwise separable convolution structure, the module can significantly reduce the computational complexity. Inside the module, a multi-branch bottleneck design is adopted, which includes four parallel sub-modules: two 3x3 depth convolution branches, mainly used to capture local spatial details and mine subtle spatial structure information in the feature map. Two 1x1 convolution branches are used for cross-channel information interaction, realizing feature fusion and transformation between channels. These sub-modules cooperate with group normalization (GN) operations to alleviate the problem of internal covariate shift, and then introduce nonlinearity through the ReLU activation function, so as to realize efficient feature enhancement. Finally, the original input information and the processed feature information are added through a residual connection to obtain the output, which can not only retain the original information and effectively avoid the problem of gradient disappearance, but also help the model to learn more rich feature differences. As shown in FIG. 2. Figure 7

[0087] The SAVSS_2D module has stronger direction perception by modeling different directions in sequence to improve long-distance information interaction efficiency. First, the enhanced feature map processed by the GBC module is unfolded from the spatial dimension to the sequence form as the input of the SAVSS_2D module. The SAVSS_2D module processes the input features by combining the structured state space model (SSM) with the multi-direction scanning strategy. The multi-direction scanning strategy includes horizontal scanning, vertical scanning, and double diagonal line scanning. The horizontal scanning processes the feature sequence row by row, aiming to capture the long-range dependencies between pixels in the row direction of the feature map; the vertical scanning processes the feature sequence column by column, which is used to mine the long-range dependencies in the column direction; the double diagonal line scanning scans the feature map from two diagonal directions to obtain the correlation information between pixels in the diagonal direction. During the scanning process, the module dynamically generates transition matrices and projection weights according to the input features. These parameters are adaptively adjusted according to the input features, so that the state transition process can better adapt to the changes of the features, thereby improving the modeling ability of complex spatial patterns. For each scanning direction, there is a corresponding structured state space model. Based on the input feature sequence, transition matrix and projection weight, the state transition rule is calculated. Finally, the outputs of the four scanning paths are dynamically weighted and fused. The weight for each scanning direction output is calculated and satisfies the condition that the sum of the weights is 1. Through weighted summation, the final serialized feature is obtained.

[0088] ​The PAF module improves the feature fusion performance through dynamic position perception fusion. The serialized features output by the SAVSS_2D module and the original spatial features processed by the GBC module are taken as inputs. Through double feature transformation, the original spatial features are converted into base features, and the serialized features are converted into guide features and key-value pairs are generated. Based on these key-value pairs, a spatial similarity graph is calculated, which can quantify the correlation of features in spatial position. Then, the original spatial features and SSM features are soft-assigned and fused with the spatial similarity graph as the weight, and finally the fused features are obtained. This fusion method can not only retain the bottom-level detail information of the original spatial features, but also inject the global structured information of the serialized features, which is suitable for the feature complementarity of small target behaviors in classroom images. Figure 9

[0089] Step four, use the training set and the validation set to train the classroom behavior recognition detection model, use the test set to evaluate the performance of the classroom behavior recognition detection model, and get the optimal classroom behavior detection model.

[0090] The training set and the validation set are input into the classroom behavior recognition detection model based on the improved YOLOv12 model, and the number of training times is set. As the number of training times increases, the loss function curve of the model gradually converges. When the loss function curve converges and stabilizes, the model is trained to the optimal state, and the optimal model weight file is saved. The test set is input into the trained optimal classroom behavior recognition detection model, and the optimal classroom behavior recognition detection image is output. According to the detection image, the performance of the classroom behavior recognition detection model based on the improved YOLOv12 model is evaluated, and the optimal detection model is obtained.

[0091] Step five, combine monitoring and computer algorithms to perform real-time recognition and detection of student classroom behavior.

[0092] In the classroom environment, arrange high-definition monitoring cameras reasonably to ensure that they can cover the students' learning area in all directions and without dead angles, capture key information such as students' behavior, posture, and facial expressions from different angles, and optimize the resolution, frame rate, and other parameters of the cameras in advance according to the size of the classroom, light conditions, and the required detection behavior to ensure the clarity and continuity of the collected images, providing high-quality raw data for subsequent accurate recognition. Through wired or wireless transmission, the video stream data collected by the monitoring cameras is transmitted stably and quickly to the pre-configured computer processing terminal. After receiving the video stream data, the computer terminal immediately starts the trained optimal classroom behavior detection model based on the improved YOLOv12, and quickly extracts and analyzes the features of each input frame of image. Once the detection model identifies that a student has a typical classroom behavior, it will be fed back in a visual way on the computer display interface in real time.​

Claims

1. A classroom behavior recognition method based on an improved YOLOv12 model, characterized by The method comprises the following steps: Step 1: Obtain a dataset of classroom behavior images; Step 2: Divide the dataset into validation set, training set, and test set, and process them into the network model recognition format of YOLOv12; Step 3: Use the classroom behavior recognition detection model training dataset of the improved YOLOv12 model. The improved YOLOv12 model adds a P2 detection head to the detection layer, designs and uses the A2C2f_FRFN module that combines the A2C2f with the feature refinement feedforward network FRFN module in the backbone network and the neck network, and designs and uses the C3k2_SAVSS module that combines the C3k2 with the structure perception visual state space module SAVSS in the backbone network and the neck network; The implementation process of the A2C2f_FRFN module is as follows: Input feature vector First, it is imported into the initial convolution layer built based on the parent class initialization logic. This layer implements channel compression on the input feature map through low-rank mapping, and uses the sliding convolution of the convolution kernel in the feature map space domain to perform multiplication and accumulation operations on the weights and the elements at the corresponding positions, completing the dimensional transformation of the input channel to the hidden channel and extracting feature information with basic representation capabilities; then it enters the loop processing flow composed of the ABlock_FRFN module, and passes the end element of the list as input to each module sequence in turn; the feature vector generated by the ABlock_FRFN module is processed With the original input feature vector Perform residual mapping fusion and finally output fusion features , its forward propagation mathematical expression is: ； The implementation process of the C3k2_SAVSS module is as follows: the previous level input feature Adjust the number of channels through the initial convolution layer to generate hidden features , where the number of hidden channels is ,Will Input is sent to a sequence of n SAVSS modules, each SAVSS performs a specific transformation on the features and outputs the intermediate features ; Finally, the final output Compared with the original previous level input features Add together to get the fusion result , its forward propagation mathematical expression is: The implementation process of the SAVSS module is as follows: First, the input features of the previous level are Two GBC operations are performed. The module contains four parallel submodules: two 3×3 depth convolution branches are used to capture local spatial details, two 1×1 convolution branches are used for cross-channel information interaction, and group normalization and ReLU activation functions are used to achieve efficient feature enhancement. Finally, the residual connection is used to retain the original information to avoid gradient disappearance. The enhanced feature map X gbc Expanded from the spatial dimension into a sequence form and fed into the SAVSS_2D module, the long-range dependency modeling is achieved through the structured state space model combined with the multi-directional scanning strategy; the serialized features of the SSM output and Through deep fusion of PAF module, the module first generates key-value pairs of base features and guide features through two-way feature transformation, and calculates the spatial similarity graph. To quantify the position correlation, The original features and SSM features are soft-assigned and fused for weights. In the final stage, linear transformation and residual connection are used to stabilize the training process: group normalization alleviates internal covariate shift, linear layer improves feature dimension adaptability, and jump connection ensures lossless information transmission and output. , its forward propagation mathematical expression is: Where σ is the Sigmoid function, ⊙ represents element-by-element multiplication; Step 4: Use the training set and validation set to train the classroom behavior recognition detection model. After obtaining the optimal detection model, use the test set to evaluate the performance of the classroom behavior recognition detection model. Step 5: Combine monitoring with the optimal detection model to conduct real-time identification and detection of students' classroom behavior.

2. The classroom behavior recognition method based on the improved YOLOv12 model according to claim 1 is characterized in that The implementation process of the ABlock_FRFN module is as follows: the previous level input feature vector First, it is processed by the regional attention module inherited from the ABlock class; the regional attention mechanism will assign weights to the features, highlight key information, and obtain the feature vector , and the original previous level input feature vector Perform the first residual connection and get ; The multi-layer perceptron module passed into the parent class ABlock performs feature transformation to obtain the feature vector , and with Perform the second residual connection and get ; Enter the FRFN2D module for processing, and finally obtain the feature vector and Perform residual connection to obtain output Output, and its forward propagation mathematical expression is: 。 3. The classroom behavior recognition method based on the improved YOLOv12 model according to claim 2 is characterized in that The FRFN2D module implementation process is as follows: the previous level input feature vector First, it is divided into two parts according to the channel dimension and ,right Perform 3×3 convolution and then The spliced ​​features are spliced ​​along the channel dimension, and the number of channels is expanded by linear1. Then, the features are divided into two parts. and ,right Perform depth-wise separable convolution and then Multiply element by element, and finally, restore the number of channels to the input dimension dim through linear2. Without losing key information, integrate and compress the features to obtain the final output Output. Its forward propagation mathematical expression is: 。 4. The classroom behavior recognition method based on the improved YOLOv12 model according to claim 1 is characterized in that The implementation process of the GBC module is as follows: the original upper level input feature To perform GBC operations, this module adopts a multi-branch bottleneck design and contains four parallel submodules: two 3×3 depthwise convolution branches for capturing local spatial details and mining subtle spatial structure information in feature maps, and two 1×1 convolution branches for cross-channel information interaction to achieve feature fusion and transformation between channels. These submodules cooperate with group normalization operations to alleviate the problem of internal covariate shift, and then introduce nonlinearity through the ReLU activation function to achieve efficient feature enhancement; finally, the original input information and the processed feature information are added together through the residual connection to obtain the output, where is the enhanced feature map after two GBC operations. The first GBC operation obtains , the forward propagation mathematical expression is: 。 5. The classroom behavior recognition method based on the improved YOLOv12 model according to claim 1 is characterized in that The implementation process of the SAVSS_2D module is as follows: First, the enhanced feature map processed by the GBC module is Expanded from the spatial dimension into a sequence form as the input of the SAVSS_2D module; the SAVSS_2D module uses a structured state space model combined with a multi-directional scanning strategy to process the input features. The multi-directional scanning strategy includes horizontal scanning, vertical scanning, and dual diagonal scanning. The horizontal scanning processes the feature sequence row by row to capture the long-range dependency between pixels in the row direction of the feature map; the vertical scanning processes the feature sequence column by column to mine the long-range dependency in the column direction; the dual diagonal scanning scans the feature map from two diagonal directions to obtain the correlation information between pixels in the diagonal direction; during the scanning process, the module dynamically generates a transfer matrix based on the input features and projection weights , for each scanning direction , have corresponding structured state space models , based on the input feature sequence, transfer matrix and projection weight, the calculation is performed through the state transition rule; finally, the outputs of the four scanning paths are dynamically weighted and fused to calculate the weight of the output for each scanning direction , and satisfy the condition that the weight sum is 1, and the final serialized feature is obtained by weighted summation , its forward propagation mathematical expression is: 。 6. The classroom behavior recognition method based on the improved YOLOv12 model according to claim 1 is characterized in that The PAF module implementation process is as follows: The PAF module uses the serialized features output by the SAVSS_2D module And the original spatial features after processing by the GBC module As input, through two-way feature transformation, Convert to base features, Convert it into guided features and generate key-value pairs, based on which the spatial similarity graph is calculated. , and then is the weight of the original feature and SSM characteristics Perform soft assignment fusion and finally obtain the fused features , its forward propagation mathematical expression is: 。 7. The classroom behavior recognition method based on the improved YOLOv12 model according to claim 1 is characterized in that The specific steps of step five are as follows: Step 5: Place high-definition surveillance cameras in the classroom environment to capture key information about students' movements, postures, and facial expressions from different angles. Step 52: Transmit the video stream data collected in real time by the surveillance camera to the computer processing terminal; Step 5.3: After the computer terminal receives the video stream data, it uses the optimal detection model obtained in step 4 to extract and analyze the features of each input frame image. Once the optimal classroom behavior detection model identifies that the student has typical classroom behavior, it will provide real-time feedback in a visual manner on the computer display interface.

Images