Examination room abnormal behavior detection method based on multilevel feature fusion

Abstract

The invention discloses a multilevel feature fusion examination room abnormal behavior detection method and device. The method comprises the following steps: constructing a YOLO-EF network model, introducing an RFConv module into a backbone network, and replacing convolution in a Bottleneck structure with RFConv to enhance feature extraction and context sensing capabilities; in the feature fusion stage, a CGAF mechanism is introduced, and dynamic weighted fusion is carried out according to semantic importance of different scale features, so that the detection performance of a long-distance and small-scale target is improved; meanwhile, a GEIT module is provided, joint optimization is carried out on a network shallow layer, a down-sampling path and a feature fusion layer, and the edge feature capturing capability of the network is enhanced. According to the invention, the accuracy and robustness of abnormal behavior detection in an examination room monitoring scene can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of target detection and computer vision, and particularly relates to a multi-level feature fusion examination room abnormal behavior detection technology BACKGROUND

[0002] In recent years, with the rapid development of computer vision technology, target detection has become a key research direction in this field. The goal of target detection is to identify and locate specific target objects from images or videos. This technology not only has significant applications in automatic driving, video surveillance, face recognition, and intelligent retail, but also has important value in the field of education, especially in examination room behavior analysis and intelligent monitoring. With the deepening of education informatization and intelligentization, examination room management is gradually transforming towards digitization and intelligentization. How to use visual perception technology to accurately identify and analyze the behavior of examinees in real time has become one of the key problems in the intelligent education and examination safety system.

[0003] In traditional offline examinations, the work of invigilators mainly relies on visual observation, through patrolling the examination room and observing the behavior of examinees to ensure the fairness and standardization of the examination. However, with the expansion of the scale of the examination room and the increase in the number of examinees, the number of invigilators often cannot meet the needs. At this time, the efficiency and accuracy of manual invigilation are severely tested. Invigilators have limited energy and cannot concentrate for a long time, which cannot achieve full-time and real-time monitoring of the examination room. In addition, the complex and diverse environment in large-scale examination rooms also makes manual invigilation inefficient, and it is easy to miss detection, which cannot effectively guarantee the fairness and standardization of the examination room.

[0004] Under this background, the examination room abnormal behavior detection technology based on multi-level feature fusion has gradually become a research hotspot. This technology can detect and identify the multi-class behavior of examinees (such as leaving the seat, raising the hand, communicating, or other abnormal actions) in the examination room through automated and intelligent means, providing technical support for examination safety management, behavior analysis, and abnormal event warning. Deep learning, especially convolutional neural networks (CNN), can automatically extract feature information from different levels through learning from large-scale image or video data, thereby significantly improving the accuracy and robustness of behavior detection. Compared with traditional methods that rely on manual feature design and rule setting, deep learning models can achieve multi-scale feature extraction from low-level to high-level semantics, better adapting to complex and variable examination room environments.

[0005] At present, the target detection algorithm based on deep learning mainly includes two-stage detection method and single-stage detection method. The two-stage detection method (such as R-CNN, Fast R-CNN, Faster R-CNN) generates candidate regions through a region proposal network (RPN), and then performs feature extraction and classification, has high detection accuracy, and is suitable for accurate identification of subtle actions or local abnormal behaviors. The single-stage detection method (such as YOLO series, SSD, etc.) realizes fast detection in an end-to-end structure, which can meet the needs of real-time monitoring in the examination room while ensuring high accuracy.

[0006] However, the traditional single-stage or two-stage detection framework still has limitations in feature expression. Since the features of different network layers differ in semantic level and spatial resolution, directly using single-layer features often cannot balance the detection accuracy. In view of this problem, a multi-level feature fusion strategy is introduced into the examination room abnormal behavior detection task, which fuses semantic features and spatial information of different levels, so that the model can capture the global posture change of the examinee and pay attention to the local detail action, thereby significantly improving the detection accuracy and abnormal behavior recognition ability. The detection method based on multi-level feature fusion can realize accurate identification of abnormal behaviors of examinees in complex examination room scenes, and provides a new technical path for the construction of intelligent examination room monitoring system. SUMMARY

[0007] To solve the problems of insufficient detection accuracy of examination room abnormal behavior and weak recognition ability for long-distance and occluded examinees in the prior art, the first aspect of the present application provides a multi-level feature fusion examination room abnormal behavior detection method, which is aimed at the identification needs of examination room abnormal behaviors, and the method comprises:

[0008] The YOLO11 basic network is improved to obtain a YOLO-EF (EdgeFusion) network, which is composed of a backbone network, a neck network and a detection head. The YOLO11 basic network refers to the basic architecture of the 11th version of the YOLO algorithm series based on the present application. The YOLO-EF network is an improved network model proposed by the present application, which integrates RFConv, C3k2_RFConv, GEIT and CGAF modules. The backbone network uses RFConv modules to replace Conv, uses RFConv to replace part of Conv in the C3k2 module bottleneck, adds MSEG modules and CEF modules in the shallow layer of the backbone. The neck network uses C3k2_RFConv to replace the original C3k2 module, and adds CEF modules and CGAF modules.

[0009] RFConv module: the application proposes a convolution structure based on local receptive field weight generation and dynamic weighting fusion. The module has two branches, the input feature map is first passed through the AvgPool and the branch of group convolution (Group convolution) to generate dynamic weights; at the same time, the input is also passed through another convolution branch, which is through the channel-by-channel convolution operation with batch normalization and nonlinear activation, to enhance the local features and spatial reconstruction of the input feature map, and multiply the dynamic weights generated by the first branch and the features generated by the second branch by position to realize the adaptive adjustment of the convolution kernel weight; finally, the standard convolution is used for feature integration and output.

[0010] C3k2_RFConv module: the application proposes a feature enhancement module integrating RFConv based on the multi-branch bottleneck stacking structure. The module introduces the RFConv module for enhancing feature expression on the basis of maintaining the original multi-branch bottleneck stacking structure, and replaces part of the Conv in the bottleneck with the above-mentioned RFConv. Among them, the RFConv module is used to dynamically adjust the receptive field range of the input feature map, and the multi-layer stacked bottleneck_RFConv structure further strengthens the interaction of the feature layer, realizing the global perception of the deep feature.

[0011] MSEG module: the application proposes a structure for generating multi-scale edge features. The module sends the input feature map into SobelConv to extract edge information, obtains gradient changes to form an edge feature map through a Sobel operator, and generates edge representations of different scales by performing multiple maximum pooling on the edge features. Each scale of feature is adjusted in channel number by 1x1 convolution, so that the multi-scale edge features remain consistent in channel space. Finally, a feature set containing edge information of different resolutions is obtained, which is used for subsequent fusion and feature enhancement.

[0012] SobelConv module: based on the fixed direction gradient kernel of Sobel operator to realize edge feature extraction. The horizontal and vertical Sobel convolution kernels are constructed inside, and are respectively applied to each channel of the input feature map in the form of depth separable three-dimensional convolution, to realize independent detection of local gradient changes.

[0013] CEF module: the application proposes a structure for channel-level fusion and edge feature enhancement of multi-source features. The input multi-scale or multi-branch features are spliced in the channel dimension, and a 1x1 convolution is used to realize channel compression and feature fusion. Then a 3x3 convolution is used to further extract local spatial features and enhance edge response, and a 1x1 convolution is used to complete feature reorganization and output channel adjustment.

[0014] CGAF module: a multi-level attention fusion structure proposed by the application. The module superimposes the two input features to form an initial fusion feature, and extracts multi-level saliency information through channel attention (CA), spatial attention (SA) and pixel attention (PA) branches respectively. Among them, CA is used to model the global dependence between different feature channels to highlight the important feature dimensions in semantics; SA focuses on mining the spatial response features of the salient region to enhance the attention of the model to the key behavior region; PA performs pixel-by-pixel adaptive weighting on the feature map under the guidance of the channel and space generated by the former two, to realize more fine local feature enhancement.

[0015] Compared with the prior art, the method of the application uses a variety of feature extraction and fusion modules to enhance the perception ability of the target detection model to edges, local details and multi-source features, and improve the discriminability and robustness of feature expression. The application is helpful to improve the overall detection accuracy and robustness of the model.

[0016] Other advantages, objects and features of the application will be partly embodied by the following description, and partly understood by those skilled in the art through research and practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An exam abnormal behavior detection method using multi-level feature fusion of the application;

[0018] Figure 2 A network structure diagram of the detection method of the application;

[0019] Figure 3 RFConv module in the detection method of the application;

[0020] Figure 4 C3k2_RFConv module in the backbone network in the detection method of the application;

[0021] Figure 5 MSEG module in the backbone network in the detection method of the application;

[0022] Figure 6 CEF module of the backbone network in the detection method of the application;

[0023] Figure 7 CGAF module of the neck network in the detection method of the application;

[0024] Figure 8 Detection effect diagram of the exam abnormal behavior detection method of the application; DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0026] Figure 1 An exam room abnormal behavior detection method based on multi-level feature fusion of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the method comprises the following steps:

[0027] S1, frame processing is performed on the exam room monitoring video recording, and an exam room abnormal behavior data set is extracted.

[0028] S2, the obtained exam room abnormal behavior data set is screened and data labeled. The exam room abnormal behavior data set is labeled by using LabelImg software, and is divided into 9 categories, which are: student answering, student non-answering, student head deviation, student standing, teacher standing, teacher sitting, student raising hand, student picking up things, and teacher distributing test papers. A data label in YOLO format is adopted, which contains label information of a target frame. After labeling, the training set and the test set are divided according to a ratio of 8:2.

[0029] S3, the YOLO11 basic network structure is improved to obtain a YOLO-EF network.

[0030] S4, the YOLO-EF network structure is used to train the exam room abnormal behavior data set, and a YOLO-EF network model is obtained.

[0031] S5, the YOLO-EF network model is used to detect the exam room abnormal behavior pictures, and the exam room abnormal behavior is effectively detected.

[0032] The YOLO-EF network structure obtained in S3 is shown in FIG. 2. Figure 2 As shown in FIG. 2, the Conv module in the backbone network is replaced by the RFConv module, and the MSEG and CEF modules are added. The CGAF module is added to the neck network.

[0033] The module structure of the RFConv is shown in FIG. 3. Figure 3 As shown in FIG. 3, the calculation can be represented as:

[0034] F = Softmax(g 1×1 (Avgpool(X))) x ReLU(Norm(g k×k (X)))

[0035] Where g1×1 represents a 1×1 grouped convolution, k represents the size of the convolution kernel (3 is used in this invention), Norm represents normalization, Avgpool represents average pooling, ReLU represents the Rectified Linear Unit activation function, Softmax represents the normalization function, X represents the input feature map, and F is the output feature obtained by multiplying the attention map with the transformed receptive field spatial features.

[0036] The C3k2_RFConv module structure is referenced. Figure 4 As shown, the calculation can be expressed as:

[0037] X1, X2 = Split(X)

[0038] X′2=Y(X2)

[0039]

[0040] F = Conv 1×1 (Concat(X1,X′2))

[0041] Where X represents the input feature map, Split represents dividing the input feature map into two equal-channel branches along the channel dimension, labeled X1 and X2 respectively, and Y contains two operations, selected according to the value of C3k (true or false): when C3k = true, the composite convolution module C3k_RFConv based on the improved C3k structure is used; when C3k = true, the composite convolution module C3k_RFConv is used; when C3k = true, the composite convolution module C3k_RFConv is used.

[0042] When false, an improved bottleneck structure, Bottleneck_RFConv, is used, embedding RFConv. n represents the number of iterations, Concat represents concatenating features along the channel dimension, and Conv... 1×1 This is represented as a 1×1 convolution operation, where F represents the final output feature map.

[0043] The C3k2_RFConv module introduces the RFConv module during deep feature extraction. First, it uses 1×1 convolutions to perform channel compression and separation operations on the input features, generating two feature branches, X1 and X2. The main branch X2 performs deep feature modeling through several stacked modules composed of Bottleneck_RFConv or C3k_RFConv. This stacked structure introduces adaptive RFConv operations at the bottleneck layer, enabling the network to maintain the local convolutional receptive field while possessing global dynamic feature response capabilities. Each Bottleneck_RFConv module internally adopts a "Conv–RFConv–residual addition" structure, while C3k_RFConv further introduces multiple layers of Bottleneck_RFConv to further enhance feature representation capabilities, particularly for small targets far from the camera.

[0044] MSEG module structure, refer to Figure 5 As shown, the calculation can be expressed as:

[0045] E x =S x *x,E y =S y *x,E=E x +E y

[0046] E i =P (i) (E), i = 1, 2, ..., n,

[0047]

[0048] MSEG(x) = [y1, y2, ..., yn]

[0049] Where x is the input feature map, S x and S y Sobel convolution kernels representing the horizontal and vertical edges, E x and E y Let E represent the horizontal and vertical edge responses, P represent the combined edge features, Ei represent the max pooling operation, and yi represent the edge features at the i-th scale.

[0050] The MSEG module uses Sobel convolution to perform edge detection on the feature map of the input abnormal behavior image in the examination room, generating an initial edge response that highlights the human contour and pose boundary information. Then, it performs multiple cascaded max pooling operations on this edge map to construct a multi-scale edge feature pyramid, and uses 1×1 convolution to adapt the features at each scale to different channels, ultimately outputting a multi-scale edge information set. In the scenario of abnormal behavior detection in the examination room, this module can effectively capture the multi-scale contour changes corresponding to the postures of teachers or students (such as tilting their heads, raising their hands, standing, etc.), providing a structured and hierarchical boundary prior for subsequent behavior discrimination. Specifically, the Sobel convolution uses two fixed and non-learnable 3×3 directional gradient kernels (detecting horizontal and vertical edges respectively), independently calculating and fusing the gradient magnitude for each channel, efficiently extracting significant edge structures in the image. This operation highlights the boundary contrast between the target and the desktop, chair, and adjacent seats, strengthens subtle but crucial changes in limb displacement or occlusion, and improves the model's sensitivity to key boundary position information.

[0051] CEF module structure, refer to Figure 6 As shown, the calculation can be expressed as:

[0052] F = f 1×1 (f 3×3 (f 1×1 (Concat(X1,…,Xn))))

[0053] Where X1,…,Xn represent the input multi-scale feature maps, Concat represents concatenating the features along the channel dimension, f represents the convolution operation, and F is the output feature map.

[0054] The CEF module concatenates multiple input feature maps along the channel dimension; it then performs channel compression and cross-channel information fusion using a 1×1 convolution, mapping the data to the intermediate channel dimension; next, a 3×3 convolution extracts local spatial context features, enhancing the structural details of the fused data; finally, a 1×1 convolution adjusts the number of channels to the target output dimension, completing feature reconstruction. This operation effectively fuses edge cues at different scales to improve the perception of subtle pose changes.

[0055] CGAF module structure, refer to Figure 7 As shown, the calculation can be expressed as:

[0056] sum = x + y

[0057] W=Sigmoid(PA(sum,SA(sum)+CA(sum)))

[0058] F = Conv(W*x + (1-W)*y + sum)

[0059] In this diagram, `sum` represents the element-wise summation of the input feature maps `x` and `y`, indicating the initial fused feature maps. `CA` represents channel attention, `SA` represents spatial attention, and `PA` represents pixel attention. The output `W` is a pixel-level weight map used to adjust the contribution of each pixel in `x` and `y`. `Sigmoid` is the activation function, `Conv` is the convolution operation, and `F` is the final output feature map.

[0060] The CGAF module adds the two input features element-wise to obtain the initial fused feature. It then extracts the channel weights and spatial weights of this initial feature using both channel and spatial attention modules, and adds them together to form a first-level pixel guidance map. Finally, it combines the initial feature with this guidance map... Figure 1 The inputs are fed into a pixel attention module to generate refined pixel-level fusion weights. These weights are then used to perform a weighted fusion of the two original inputs, which is then added to the initial fusion result and output as the final feature through a 1×1 convolution. This module can dynamically balance the importance of features from different sources and adaptively emphasize behavior-related salient regions and channel information.

[0061] In step S4, the YOLO-EF network is trained using the obtained abnormal behavior dataset for the examination room to obtain the optimal model for detecting abnormal behavior in the examination room. In this embodiment, the device uses an NVIDIA GeForce RTX 4070TiSUPER graphics card, the model uses the PyTorch framework, the batch size is 32, the epochs are 300, the optimizer is SGD, the initial learning rate is 0.01, and the momentum size is 0.973.

[0062] In step S5, the trained YOLO-EF model is used to detect abnormal behavior images in the examination room. The accuracy is better than the YOLO11 basic model, which improves the generalization and robustness of the model and effectively detects abnormal behavior cases in the examination room.

[0063] This invention achieves efficient feature extraction and multi-scale fusion for detecting abnormal behavior in examination rooms by systematically improving the YOLO11 network. Specific improvements include: replacing traditional convolutional operations with RFConv in the backbone network and constructing a new C3k2 module using RFConv in the Bottleneck structure, thereby enhancing the network's ability to express fine-grained behavioral features and local action patterns; introducing a CGAF module into the feature fusion network, achieving adaptive weighted fusion of multi-source features at the pixel level through the synergistic effect of channel attention, spatial attention, and pixel-level attention, enabling the network to automatically focus on the key action regions of the examinee and suppress background interference; and introducing a GEIT module into the Backbone and Neck parts to improve the interaction efficiency and information transmission capability of multi-scale features, enhancing the ability to capture behavioral features in complex scenes. Experimental results show that the improved YOLO-EF network model outperforms the basic YOLO11 version in all detection metrics on the examination room abnormal behavior dataset. Specifically, mAP@0.5 is improved by 3.2%, and mAP@0.5:0.95 is improved by 3.3%. On the Visdone2019 dataset from a drone's perspective, YOLO-EF also demonstrates significant advantages in detecting small and occluded targets, with mAP@0.5 improved by 4.2% and mAP@0.5:0.95 by 3.1% compared to the basic YOLO11 version. These results fully demonstrate that the improved YOLO-EF network significantly enhances the ability to identify distant and occluded examinees, making it suitable for abnormal behavior detection in examination room monitoring scenarios. Model metrics are shown in Tables 1 and 2, and some detection results are referenced... Figure 8 As shown.

[0064] Table 1. Dataset of Abnormal Behaviors in Examination Rooms

[0065]

[0066] Table 2. Visdone 2019 Dataset

[0067]

[0068] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-level feature fusion examination room abnormal behavior detection method, characterized in that, Comprise: Backbone: replace Conv with RFConv, replace C3k2 with C3k2_RFConv; introduce GEIT module; Neck: introduce GEIT module; introduce CGAF module before Head; Wherein, the RFConv, C3k2_RFConv, GEIT, CGAF are structural features disclosed for the first time by the application.

2. The multi-level feature fusion proctor abnormal behavior detection method of claim 1, wherein, The RFConv further comprises: Local weight generation unit: performing kxk average pooling on the input feature map, and generating k2 weights corresponding to each channel through group 1x1 convolution consistent with the channel number, and performing Softmax normalization in the k2 dimension. Local feature expansion unit: using kxk group convolution to expand the input feature map, and then sequentially passing through batch normalization and activation processing to obtain k2 local sub-features of each channel. Weighted fusion unit: element-wise multiplication of the local sub-features and the corresponding normalized weights, and rearranging the kxk spatial layout to form a weighted feature map. Using kxk stride k convolution to downsample and channel fuse the weighted feature map to obtain the final output feature. Wherein, k=3, the number of group convolution groups is the same as the number of input channels.

3. The multi-layered feature fusion proctor abnormal behavior detection method of claim 1, wherein, The C3k2_RFConv further comprises: Bottleneck feature transformation unit (Bottleneck_RFConv): the input feature is first compressed in channel through 1x1 convolution, and then locally enhanced in space through embedded RFConv. Stacked intermediate module unit (C3k_RFConv): the bottleneck structure is stacked according to a preset repetition number n, and a single bottleneck structure or a C3k_RFConv structure composed of multiple bottleneck units is selected according to the configuration. Double-branch residual fusion unit: the input feature is divided into a main branch and a short-circuit branch, the main branch is connected in series with the stacked intermediate module unit, and the short-circuit branch is kept linear mapping; the two branches are finally spliced and fused in the channel dimension. Wherein, the stacking number n and the bottleneck channel compression ratio can be configured according to the network depth and the input feature channel number; the bottleneck channel compression ratio refers to the ratio of the compressed channel number to the input channel number in the bottleneck structure (Bottleneck_RFConv); the combination of the Bottleneck_RFConv and the C3k_RFConv constitutes the C3k2_RFConv module disclosed for the first time by the application.

4. The multi-layered feature fusion proctor abnormal behavior detection method of claim 1, wherein, The GEIT further comprises: SobelConv unit: convolving the input feature map along the horizontal direction and the vertical direction respectively, and adding the results element by element to obtain the basic edge feature. Multi-scale edge generation unit (MSEG): based on the edge feature output by the SobelConv unit, multiple maximum pooling is used to form different scale features, and 1x1 convolution is used to unify the channel dimension to generate a multi-scale edge feature set. Edge fusion convolution unit (CEF): concatenating multi-scale edge features in channel dimension, sequentially passing through 1x1 convolution channel fusion, 3x3 convolution spatial feature extraction and 1x1 convolution compression reconstruction to generate the fused edge perception feature map. The combination of SobelConv, MSEG and CEF constitutes the GEIT module disclosed for the first time in the present application.

5. The multi-level feature fusion proctor abnormal behavior detection method of claim 1, wherein, The CGAF further comprises: Channel attention unit (CA): performing global average pooling on the input feature map, and generating channel dimension weight through two layers of 1x1 convolution and ReLU activation. Spatial attention unit (SA): calculating the average and maximum value of the input feature map in the channel dimension respectively, and generating a spatial attention map through 7x7 convolution after splicing. Pixel attention unit (PA): pixel-by-pixel fusion of the features generated by the channel attention and the spatial attention, and generating the final pixel-level weight through grouped convolution and Sigmoid activation. Feature fusion unit: weighting and multiplying the input feature and the pixel-level weight, and further integrating the channel information through 1x1 convolution to generate the final output feature map. The specific implementation of the channel attention, spatial attention and pixel attention is not limited to the structure shown in the above embodiment, as long as the adaptive weight distribution of the input feature can be realized in the channel dimension, spatial dimension and pixel level, which can be regarded as an equivalent replacement scheme of the CGAF module of the present application.

Images