An examination room abnormal behavior detection method based on an improved YOLOv7 model

Abstract

The application discloses an examination room abnormal behavior detection method based on an improved YOLOv7 model, comprising the following steps: S1, frame processing is performed on an existing examination room monitoring video to obtain examination room picture data; S2, the examination room picture data are sequentially preprocessed and labeled to obtain an examination room abnormal behavior data set; S3, an improved YOLOv7 network model is constructed, an activation function SiLU is replaced by FReLU, a Gn-ASPPCSPC module is used to replace an ASPPCSPC module, a Gn-MP module is used to replace an MP module, and a SIoU loss function is used; S4, the improved YOLOv7 model is trained by using the examination room abnormal behavior data set, and an optimal performance model is obtained after training; and S5, the optimal performance model is used to identify newly obtained examination room pictures to detect abnormal behaviors appearing in the examination room. The detection method solves the problems of the original YOLOv7 network model, such as lack of ability to capture feature space correlation and inaccurate target positioning in a complex background.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for detecting abnormal behavior in examination rooms based on an improved YOLOv7 model. Background Technology

[0002] With the advent of the artificial intelligence era, machine vision and deep learning technologies are being applied across various fields. Currently, detecting abnormal student behavior in examination rooms requires more than just traditional invigilator patrols; it necessitates practical and effective technological means to comprehensively assess the unusual behavior of each examinee. However, most current video surveillance systems in examination rooms operate in a traditional mode, primarily focusing on recording and storage. They can only record and store simple video footage of the examination environment, failing to effectively detect abnormal behavior and relying solely on manual observation and judgment of the surveillance video. This manual review of the video recordings to identify abnormal behavior is hampered by a large amount of irrelevant and redundant storage, resulting in low efficiency in identifying student abnormal behavior.

[0003] Alexey Bochkovskiy et al. proposed the YOLOv7 algorithm, which further optimizes YOLOv5 and has significant advantages in detection accuracy and speed. The YOLOv7 algorithm consists of four main modules: Input, Backbone, Neck, and Head. The Input module performs preprocessing operations such as Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling to crop the image to a uniform size, ensuring it meets the input requirements of the Backbone. The Backbone consists of a CBS convolutional module, an MP module, and an efficient aggregation network E-ELAN module. The Neck fusion network retains the PAFPN structure of YOLOv5, extracting three feature layers located in the middle, lower middle, and bottom layers of the Backbone, ensuring effective fusion of feature layers at different scales. Finally, the Head uses RepConv to adjust the number of channels. RepConv borrows from the structure reparameterization method of RepVGG, which has different structures in the training and inference phases. In the inference phase, the structure of RepConv is simplified without loss of accuracy.

[0004] The activation function SiLU used in the original YOLOv7 network, compared to ReLU, adds smoothness, but at the same time introduces exponential calculation, increasing computational cost, and lacks the ability to capture spatial correlations, making it spatially insensitive. Furthermore, the YOLOv7 network uses a large number of max-pooling operations during feature fusion and downsampling, which can cause the loss of some key features, leading to inaccurate target localization in complex backgrounds. Its bounding box loss function uses CIoU, whose aspect ratio describes a relative value, introducing some ambiguity. Summary of the Invention

[0005] To address the problems of high computational cost, lack of ability to capture feature space correlation, and inaccurate target localization in complex backgrounds when using the existing YOLOv7 network for abnormal behavior detection in examination rooms, this invention provides an abnormal behavior detection method based on an improved YOLOv7 model.

[0006] The present invention provides a method for detecting abnormal behavior in examination rooms based on an improved YOLOv7 model, the steps of which are as follows:

[0007] S1. Perform video frame segmentation on the existing examination room monitoring video to obtain examination room image data; the examination room monitoring video is collected by the video monitoring device.

[0008] S2. The obtained examination room image data is preprocessed and labeled sequentially to obtain the examination room abnormal behavior dataset.

[0009] The specific steps are as follows:

[0010] S21. Preprocessing: Rotate, flip, crop, and change the color saturation of the exam room images to bring the total number of images to 1800-2500.

[0011] S22. Labeling: The preprocessed dataset was labeled using Label Img software, and the student status in the examination room was divided into nine categories: sitting normally - answering questions, sitting normally - not answering questions, standing, sitting upright and looking forward, looking around significantly, looking around significantly, reaching out significantly, raising hands, and lying down.

[0012] S23. Format Conversion: The dataset labeled by LabelImg software can generate a JSON-type label file. Before it is officially put into the network for training, it needs to be converted into a format that meets the training requirements of YOLOv7 network. The converted dataset is divided into training and test sets in an 8:2 ratio.

[0013] S3. Construct an improved YOLOv7 network model: This model consists of four main modules: Input, Backbone, Neck, and Head.

[0014] In the backbone feature extraction network, the activation function SiLU in the CBS convolutional module is replaced with FReLU to form a new CBF convolutional module; the formula for the activation function FReLU is as follows:

[0015] f(x c,i,j ) = max(x c,i,j ,T(x c,i,j ))

[0016]

[0017] In the formula, T (x c,i,j ) represents the funnel-shaped condition defined, x c,i,j This represents the window centered at 2D position (i, j) on the c-th channel. This indicates the parameters shared by this window within the same channel.

[0018] The feature fusion network uses the Gn-ASPPCSPC module to replace the ASPPCSPC module and the Gn-MP module to replace the MP module.

[0019] The Gn-ASPPCSPC module retains the CSP residual structure of the original module, replaces the CBS module with the CBF module, and uses dilated convolution instead of max pooling. It samples in parallel with dilated convolutions at different sampling rates, and constructs convolution kernels with different receptive fields through different sampling rates to obtain multi-scale object information. Finally, the results from multiple branches are fused together, and recursive gated convolution is introduced to perform high-order spatial interaction on the multi-scale fused features.

[0020] The Gn-MP module has a two-branch structure. The first branch passes through a maxpooling module and then a GBF module. The second branch passes through a parameter-free attention mechanism, then through a 1x1 convolutional GBF module, and then through a 3x3 convolutional GBF module with a stride of 2. Finally, the results of the first and second branches are added together to obtain the downsampling result. The GBF module consists of recursive gated convolution, BN, and the activation function FReLU.

[0021] The detection head network uses the SIoU loss function as the localization loss function for bounding box regression. The formula for the SIoU loss function is as follows:

[0022]

[0023]

[0024]

[0025] In the formula, IoU represents IoU loss; Δ represents distance cost, which is the distance between the center points of the predicted box and the ground truth box; Ω represents shape cost; θ represents the degree of attention given to shape loss; γ represents the distance value assigned time priority; ω w and ω h The formula is expressed as follows:

[0026]

[0027]

[0028] w is the width of the prediction box, w gt h is the width of the ground truth bounding box, and h is the height of the predicted bounding box. gt The height of the actual bounding box.

[0029] S4. Use the abnormal test room behavior dataset obtained in step S2 to train the improved YOLOv7 network model. After training, the optimal performance model is obtained.

[0030] S5. The newly acquired exam room images are identified using the optimal performance model, effectively detecting abnormal behaviors in the exam room.

[0031] Compared with the prior art, the advantages of the present invention are:

[0032] The method of this invention uses the visual activation function FReLU to replace the visual extraction method of the SiLU-optimized network to capture more spatial visual information and improve recognition accuracy. In the feature fusion network, improved Gn-ASPPCSPC and Gn-MP modules are used, introducing dilated convolutions to increase the receptive field without losing information through max pooling. Recursive gated convolutions are used to achieve high-order spatial interactions of key features, and a parameter-free attention activation mechanism is used to highlight key information, achieving effective fusion of multi-scale features while reducing the loss of key features. The original CIoU loss function is replaced with the SIoU loss function, incorporating angular cost and redefining the distance using angular cost, thus reducing the total degrees of freedom of the loss function. Due to the increased angular cost, the loss function is more fully expressed, while reducing the probability of the penalty term being zero, resulting in smoother convergence of the loss function, improved regression accuracy, and thus reduced prediction error.

[0033] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0034] Figure 1 This is a flowchart of the abnormal behavior detection method in the examination room based on the improved YOLOv7 model of the present invention.

[0035] Figure 2 This is a network structure diagram of the detection method of the present invention.

[0036] Figure 3 This refers to the CBF module in the network model of the detection method of the present invention.

[0037] Figure 4 This refers to the Gn-ASPPCSPC module in the neck feature fusion network of this invention.

[0038] Figure 5 This is a flowchart illustrating the operation of the recursive gated convolution (gnconv) used in this invention.

[0039] Figure 6 This is a schematic diagram of SimAM.

[0040] Figure 7 This refers to the Gn-MP module in the neck feature fusion network of this invention.

[0041] Figure 8 This is a schematic diagram showing the parameter representation of the SiOU loss function used in the detection head network of this invention.

[0042] Figure 9 This is an example of the detection effect of some abnormal behaviors in the practical application of the method of the present invention. Detailed Implementation

[0043] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0044] like Figure 1 As shown, the examination room abnormal behavior detection method based on the improved YOLOv7 model of the present invention includes the following five steps performed in sequence:

[0045] S1. Use video surveillance devices to collect exam videos in the exam room, process the existing exam room surveillance videos into frames, and obtain exam room image data.

[0046] S2. The obtained examination room image data is preprocessed and labeled sequentially to obtain the examination room abnormal behavior dataset.

[0047] The obtained exam room image data was preprocessed and labeled to obtain an exam room abnormal behavior dataset. This included rotating, flipping, cropping, and changing the color saturation of the original images, ultimately resulting in approximately 2000 images. The preprocessed dataset was labeled using LabelImg software, categorizing student states in the exam room into nine types: sitting normally while answering questions, sitting normally while not answering questions, standing, sitting upright and looking forward, looking significantly to the left and right, looking significantly backward, reaching out significantly, raising hands, and lying down. The labeled dataset generated by LabelImg software produces a JSON-type annotation file, which needs to be converted to a format suitable for YOLOv7 network training before being used for network training. Specifically, this is done by parsing the JSON file to obtain original image information, polygon bounding box annotation information, and labeled target categories, calculating the coordinates of the minimum bounding box surrounding the target, and generating a target detection label file with the same name as the JSON annotation file but of type TXT. Subsequently, the converted dataset is divided into training and test sets in an 8:2 ratio.

[0048] S3. Construct a YOLOv7 network. During the construction process, improve the YOLOv7 network to obtain an improved YOLOv7 network model.

[0049] S4. Use the abnormal behavior dataset obtained in step S2 to train the improved YOLOv7 network model. After training, the optimal YOLOv7 network model is obtained.

[0050] S5. The optimal YOLOv7 network model is used to identify newly acquired exam room images, effectively detecting abnormal behaviors in the exam room.

[0051] The structure of the improved YOLOv7 network model obtained from step S3 is as follows: Figure 2 As shown in the diagram. The backbone feature extraction network employs a CBF convolutional module, the structure of which is shown below. Figure 3 As shown, it consists of Conv, BN, and the activation function FReLU. With only a slight increase in computational burden, ReLU and PReLU are extended into 2D activation functions. Specifically, the conditional part within the max() function (the original x<0 part of ReLU) is replaced with a 2D funnel condition, solving the spatial insensitivity problem in activation functions. This allows even regular (ordinary) convolutions to capture complex visual layouts, enabling the model to perform pixel-level modeling. The visual activation function FReLU is used to capture more spatial visual information, improving recognition accuracy.

[0052] Figure 4The Gn-ASPPCSPC module retains the CSP residual structure of the original module, replaces the CBS module with the CBF module, and optimizes the network's visual extraction method by replacing SiLU with the visual activation function FReLU to capture more spatial visual information and improve recognition accuracy. The original module used Maxpooling to increase the receptive field while reducing resolution, but this operation leads to the loss of some detail information. To reduce this loss, the new module uses dilated convolution instead of maxpooling, sampling in parallel with dilated convolutions at different sampling rates. Convolutional kernels with different receptive fields are constructed using different sampling rates to acquire multi-scale object information. Finally, the results from multiple branches are concatenated and a recursive gated convolution gnconv is introduced to perform high-order spatial interactions on the multi-scale fused features.

[0053] Figure 5 The recursive gated convolution shown is a convolutional module that is efficient, scalable, and translation-invariant. It uses gated convolution and recursive design for efficient implementation of arbitrary-order spatial interactions. Let the input features X∈R. HW×C The input feature X is processed by a linear layer to output feature maps p0 and q0. Then, p0 undergoes a depthwise convolution, followed by a dot product with q0 to obtain p1. Finally, p1 is processed by a linear layer to output y. The output of gnconv can be represented as:

[0054]

[0055] p1=f(q0)·p0∈R HW×C

[0056] y=φ(p1)∈R HW×C

[0057] Here, f represents depthwise convolution, and · represents dot product operation. By repeatedly performing gated convolution operations, higher-order interactions between features can be achieved.

[0058] The MP module of the original YOLOv7 network is used for downsampling. However, during downsampling, the feature map size shrinks, causing the loss of some key features and leading to inaccurate target localization in complex backgrounds. To address this loss of key features, this invention introduces recursively gated convolutions (gnconv) into the MP module to replace standard convolutions, enabling higher-order interactions of key features. Simultaneously, a parameter-free attention mechanism (SimAM) is introduced. Compared to channel and spatial attention mechanisms, SimAM directly infers three-dimensional attention weights within the network layers, considering the correlation between spatial and channel dimensions without adding unnecessary parameters.

[0059] SimAM principles are as follows: Figure 6As shown, SimAM assigns higher weights to neurons carrying more key information when processing visual tasks. By spatially suppressing adjacent neurons of foreign objects, it reduces the interference of complex backgrounds on target recognition and highlights the key features of the target. The improved Gn-MP module, while utilizing a parameter-free attention mechanism to highlight key target features, achieves high-order spatial interaction of key features through recursive gated convolution, significantly reducing the loss of key features during downsampling. The structure of the Gn-MP module obtained in this invention is as follows: Figure 7 As shown.

[0060] The method of this invention replaces the original network CIoU with the SIoU loss function, incorporates angular cost into the calculation, and uses angular cost to redescribe the distance, reducing the total degrees of freedom of the loss function. The parameters used in the SIoU loss function are as follows: Figure 8 As shown.

[0061] The method of this invention uses SIOU Loss as the localization loss function for bounding box regression. The SIOU loss function consists of angle loss, distance loss, shape loss, and IOU loss. Whether to use β or α minimization is determined by whether the angle is greater than 45°. The formula for calculating the angle cost is as follows:

[0062]

[0063] in,

[0064]

[0065]

[0066]

[0067] Distance cost represents the distance between the center points of the predicted bounding box and the ground truth bounding box. SIoU redefines distance cost as follows:

[0068]

[0069]

[0070]

[0071] γ=2-Λ

[0072] As α approaches 0, the contribution of distance cost decreases significantly. Conversely, the closer α is to π / 4, the greater its contribution to distance cost. As the angle increases, γ is assigned a time-priority distance value.

[0073] Shape cost is defined as follows:

[0074]

[0075] The value of θ here defines the degree of concern for shape loss. In this method, it is set to 1, which will immediately optimize the aspect ratio of a shape, thereby restricting the free movement of the shape.

[0076] In summary, the final definition of the SIoU loss function is as follows:

[0077]

[0078] As the angle cost increases, the loss function is more fully expressed, while the probability of the penalty term being zero is reduced, making the convergence of the loss function more stable, improving the regression accuracy, and thus reducing the prediction error.

[0079] In step S4, the improved YOLOv7 network model is trained using the obtained abnormal examination behavior dataset. After training, the optimal performance model is obtained and saved as best.pt. In this embodiment, the device used is NVIDIA V100, and the entire model is implemented in PyTorch. The initial learning rate is 0.01, the momentum is 0.937, the optimization strategy uses the SGD function, the weight decay is set to 0.0005, and the batch size is 32.

[0080] In step S5, the newly acquired abnormal behavior data of the examination room is identified by the optimal performance model best.pt, which effectively detects abnormal behavior in the examination room.

[0081] This invention improves the single-stage object detection algorithm YOLOv7 by introducing recursively gated convolutions. Through gated convolutions and recursive design, it performs high-order spatial interactions, offering high flexibility and customizability. It is compatible with various convolution variants, extending second-order interactions in self-attention to arbitrary orders without introducing significant additional computation, thus addressing the problem of CNNs lacking global and long-range modeling. The visual extraction method of the network is optimized by replacing SiLU with the visual activation function FReLU to capture more spatial visual information and improve recognition accuracy. The feature fusion network uses improved Gn-ASPPCSPC and Gn-MP modules, introducing dilated convolutions to increase the receptive field without losing information through max pooling. Recursively gated convolutions are used to achieve high-order spatial interactions of key features, and a parameter-free attention activation mechanism highlights key information. This reduces feature loss caused by the feature processing in single-stage object detection networks while achieving effective fusion of multi-scale features. Compared to the original YOLOv7, the algorithm combining recursive gated convolution and improved basic modules shows a 2.7% improvement in mAP@0.5 and a 2.5% improvement in mAP@0.5:0.95 on the integrated VOC2007 and VOC2012 datasets (see Table 1). The improved algorithm was applied to exam room abnormal behavior detection, and some detection results are shown below. Figure 9 As shown.

[0082] Table 1. Comparison of Experimental Evaluation Indicators

[0083]

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for detecting abnormal behavior in examination rooms based on an improved YOLOv7 model, characterized in that, The steps are as follows: S1. Perform video frame segmentation on the existing examination room monitoring video to obtain examination room image data; S2. Preprocess and label the obtained exam room image data in sequence to obtain the exam room abnormal behavior dataset; S3. Construct an improved YOLOv7 network model: In the backbone feature extraction network, the activation function SiLU in the CBS convolutional module is replaced with FReLU to form a new CBF convolutional module. In the feature fusion network, the ASPPCSPC module is replaced by the Gn-ASPPCSPC module, and the MP module is replaced by the Gn-MP module. The Gn-ASPPCSPC module retains the CSP residual structure of the original module, replaces the CBS module with the CBF module, replaces max pooling with dilated convolution, and samples in parallel with dilated convolutions at different sampling rates. Convolutional kernels with different receptive fields are constructed through different sampling rates to obtain multi-scale object information. Finally, the results from multiple branches are fused together, and recursive gated convolution is introduced to perform high-order spatial interaction on the multi-scale fused features. The Gn-MP module has a two-branch structure. The first branch passes through a maxpooling module followed by a GBF module. The second branch passes through a parameter-free attention mechanism, a 1x1 convolutional GBF module, and a 3x3 convolutional GBF module with a stride of 2. Finally, the results of the first and second branches are added together to obtain the downsampling result. The GBF module consists of recursive gated convolution, batch normalization (BN), and the activation function FReLU. The SIoU loss function is used as the localization loss function for bounding box regression in the detection head network; S4. Use the abnormal behavior dataset obtained in step S2 to train the improved YOLOv7 network model. After training, obtain the optimal performance model. S5. The newly acquired exam room images are identified using the optimal performance model, effectively detecting abnormal behaviors in the exam room.

2. The method for detecting abnormal behavior in examination rooms based on the improved YOLOv7 model as described in claim 1, characterized in that, In step S3, the formula for the activation function FReLU is as follows: In the formula, For the defined funnel-shaped conditions, This represents the window centered at 2D position (i, j) on the c-th channel. This indicates the parameters shared by this window within the same channel.

3. The method for detecting abnormal behavior in examination rooms based on the improved YOLOv7 model as described in claim 1, characterized in that, In step S3, the formula for the SIoU loss function is as follows: In the formula, IoU represents IoU loss; △ represents distance cost, which is the distance between the center points of the predicted box and the ground truth box; Ω represents shape cost; θ represents the degree of attention given to shape loss; and γ represents the distance value assigned time priority. and The formula is expressed as follows: w is the width of the prediction box, w gt h is the width of the ground truth bounding box, and h is the height of the predicted bounding box. gt The height of the actual bounding box.

4. The method for detecting abnormal behavior in examination rooms based on the improved YOLOv7 model as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Preprocessing: Rotate, flip, crop, and change the color saturation of the exam room images to bring the total number of images to 1800-2500. S22. Labeling: The preprocessed dataset was labeled using Label Img software, and the student status in the examination room was divided into nine categories: sitting normally - answering questions, sitting normally - not answering questions, standing, sitting upright and looking forward, looking around significantly, looking around significantly, reaching out significantly, raising hands, and lying down. S23. Format Conversion: The dataset labeled by LabelImg software can generate a JSON-type label file. Before it is officially put into the network for training, it needs to be converted into a format that meets the training requirements of YOLOv7 network. The converted dataset is divided into training and test sets in an 8:2 ratio.

Images