A method for identifying abnormal behavior of examinees based on human posture assessment
The YOLOv8-pose and QRABPNet model combination addresses the limitations of traditional invigilation by accurately detecting and alerting to abnormal behaviors, enhancing examination fairness and efficiency.
Patent Information
- Application Number
- CN202410545197.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-06
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-05-06
AI Technical Summary
Existing examination monitoring systems face challenges in real-time detection of abnormal behaviors due to limited invigilator resources and visual blind spots, with face recognition technology failing to effectively identify and alert on various cheating actions.
A method combining the YOLOv8-pose model and the QRABPNet multi-classification model for real-time detection of abnormal behaviors by analyzing human posture through key point recognition and classification, using multiple cameras synchronized to ensure comprehensive coverage and accuracy.
The system effectively reduces missed detections of cheating actions by providing real-time, efficient, and accurate identification and alerting invigilators to abnormal behaviors, optimizing resource use and ensuring examination fairness.
Smart Images
Figure CN118506443B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of examination monitoring based on computer vision algorithms, and in particular relates to a method for identifying abnormal behavior of examinees based on human posture assessment. Background Art
[0002] After the exam begins, invigilators play a vital role in the exam room, monitoring the behavior of candidates and ensuring the fairness of the exam. However, since the number of invigilators is usually limited and the number of candidates is large, especially in computer-based exams, monitors and other equipment may block the invigilators' line of sight, which makes it extremely challenging to monitor the entire exam room in real time. Invigilators' constant patrols in the exam room may also distract candidates from answering questions and may even miss some abnormal behaviors.
[0003] In recent years, artificial intelligence technology has achieved remarkable rapid development and has become the focus of global scientific and technological competition. From scientific research to industry, artificial intelligence is gradually penetrating into various fields, injecting new impetus into my country's economic development. Among them, with the development of computer vision technology, visual AI provides a technical basis for the intelligent invigilation of examination rooms, which can further ensure the fairness and justice of the examination. However, at present, intelligent invigilation is mainly based on face recognition technology, which cannot effectively detect some abnormal actions and cannot perform intelligent early warning push; at the same time, the existing technology also lacks specific classification of abnormal behaviors of candidates and efficient and accurate detection methods. Summary of the invention
[0004] To solve the above problems, the present invention combines the YOLOv8-pose model and the customized QRABPNet multi-classification model to detect abnormal test behaviors in real time on the video data of the examination room during the examination.
[0005] The present invention provides a method for identifying abnormal behavior of examinees based on human posture assessment, comprising the following process:
[0006] Step 1: Based on the defined frame extraction rules, the examinee image is obtained from the real-time video stream of the monitoring system;
[0007] Step 2, preprocessing the extracted candidate images to obtain standard candidate images;
[0008] Step 3, obtain the standard candidate image, and obtain the key points of the candidate's posture through the custom posture detection model, including nose, left eye left_eye, right eye right_eye, left shoulder left_shoulder, right shoulder right_shoulder, left elbow left_elbow, right elbow right_elbow, left wrist left_wrist, right wrist right_wrist, a total of 9 key points;
[0009] Step 4, obtain 9 key point data, and identify abnormal behavior names and confidence levels through the custom classification model QRABPNet model; the abnormal behavior names include 5 categories: lying on the table, standing up and peeking forward, leaning sideways and peeking left and right, placing both hands under the table, and raising hands;
[0010] Step 5: confirm whether abnormal behavior exists based on the abnormal behavior determination rules. If so, push an abnormal behavior warning.
[0011] Preferably, the n cameras in the examination room cover the entire examination room, each camera is set with synchronization time and time interval, and the network cameras in the examination room are polled 8 times every second to extract images at the same time. The images extracted each time are organized into an image group according to the order of the network cameras, which are used for the key point extraction of the examinee's posture in step 3; the 8 images extracted within 1 second form an examination room image group, which are formed into a sequence group according to the time series. The time series images ensure the continuity of the images, which are used for abnormal behavior judgment in step 5.
[0012] Preferably, the preprocessing process in step 2 includes image stitching, candidate target extraction and normalization processing in sequence; the image stitching is to merge the images of different network cameras in the examination room at the same time into one image; the candidate target extraction is to generate an image according to each examination camera position; the normalization processing is to unify the size of the acquired examination camera position images.
[0013] Preferably, the image stitching and candidate target extraction also include a process of image contrast adjustment, specifically:
[0014] S11, block division, divides the image into several small, non-overlapping area blocks;
[0015] S12, calculating the histogram of each block, and calculating the histogram for each block separately;
[0016] S13, contrast limitation, performs contrast limitation on the histogram of each block, that is, sets a maximum threshold. When the number of pixels at a certain gray level exceeds this threshold, these redundant pixels are evenly distributed to other gray levels.
[0017] S14, histogram equalization, performing histogram equalization on each block using the modified histogram;
[0018] S15, histogram accumulation, in order to avoid obvious boundaries between blocks, use gradient transition or bilinear interpolation to smooth the histogram between blocks;
[0019] S16, image reconstruction, reconstructs each block using the equalized histogram, and then merges all blocks into one image.
[0020] Preferably, the image stitching is specifically:
[0021] S21, camera calibration, calibrating each camera to obtain its internal and external parameters;
[0022] S22, synchronously collecting images to ensure that all cameras collect images at the same time point;
[0023] S23, feature extraction and matching, using SIFT feature extraction algorithm to extract feature points from the image of each camera, and feature matching algorithm to find corresponding feature points between images of different cameras;
[0024] S24, image transformation and registration, calculating the homography matrix or basic matrix between images according to the matched feature points, these matrices describe the geometric transformation relationship between images, and using these matrices to transform the images so that the images taken by different cameras can be aligned in the same coordinate system;
[0025] S25, image fusion, fusing multiple images after registration using Laplacian pyramid fusion;
[0026] S26, seam line processing, smoothing the seam line of the overlapping area after the image fusion, so that the transition is more natural.
[0027] Preferably, the candidate target extraction is to distinguish the corridor area, the vacant area, and the test camera area in combination with the test room calibration data, and generate an image according to each test camera; specifically including:
[0028] Calibration data matching: Match the calibration data with the spliced image and calculate the width and height matching coefficients (W n , H n ); take the upper left corner of the image as the reference point (TX 0 ,TY 0 ), width TW, height TX and the upper left corner of the calibration data (BX 0 , BY 0 ), width BW, height BH, the formula is as follows:
[0029]
[0030]
[0031] Image extraction: Use the matching coefficient to calculate the actual coordinates of each corridor area, vacant area, and test area, and use Opencv's getRectSubPix function to extract the sub-image from the rectangular area; calibrate the upper left corner of the sub-area (BX 0 ,BY 0 )、lower right corner (BX 1 ,BY1 ), corresponding to the upper left corner of the image coordinates (TX 0 ,TY 0 ), lower right corner (TX 1 ,TY 1 ), the calculation formula is as follows:
[0032] TX n =W n ×BX n
[0033] Among them TX n Represents the X coordinate of the calibration point to be determined, BW n Represents the original X coordinate of the calibration point;
[0034] TH n =H n ×BH n
[0035] where TH n represents the X coordinate of the calibration point to be determined, BH n Represents the original X coordinate of the calibration point.
[0036] Preferably, the normalization process is to unify the size of the acquired test camera images; including:
[0037] Calculate scaling: Calculate the aspect ratio of each test camera image, scale the image according to the ratio, and unify it into 640×480 images; during the scaling process, scale_width is the width scaling ratio, scale_height is the height scaling ratio, width is the original width of the image, and height is the original height of the image;
[0038] The image is wider than it is tall:
[0039]
[0040] Image width is less than or equal to height:
[0041]
[0042] Scale the image:
[0043] Use Opencv's resize function to scale and merge into a 640×480 image.
[0044] Preferably, the custom posture detection model in step 3 is a candidate posture model trained based on Yolov8n-pose.pt.
[0045] Preferably, the custom classification model QRABPNet model in step 4 is a multi-classification of abnormal behavior key point data; the QRABPNet model is based on the basic class nn.Module for building a neural network, and calculates the number of input features as 18 and the output category as 5 according to the coordinate data of 9 key points; first, create a fully connected layer to receive the initialized number of features, and convert the input data into a linear combination of 256 features; and use the activation function ReLU to introduce nonlinear factors to help the network learn more complex features; secondly, create a second fully connected layer of the network, the number of input features is 256, the number of output features is 51 2, and create a ReLU activation function for use after the second fully connected layer; again, create the third fully connected layer of the network, the number of input features is 512, the number of output features is 256, and create a ReLU activation function for use after the third fully connected layer; finally, by creating the fourth fully connected layer, the number of input features is 256, and the number of output features is num_classes, that is, the number of defined categories; define forward for training and reasoning to implement the forward propagation function; the QRABPNet model has a neural network with four hidden layers, and each hidden layer uses the ReLU activation function for nonlinear transformation.
[0046] Preferably, the abnormal behavior determination rule in step 5 is specifically:
[0047] Based on the classification information obtained in step 4, the analysis results of 8 images within 1 second are accumulated; the classification confidence is greater than or equal to 0.45 and is considered to be a correct anomaly; if the same type of anomaly accumulates more than 5 images within 1 second, it is confirmed that the examinee has abnormal behavior;
[0048] The analysis results of 8 images in 1 second: I 1 , I 2 , I 3 , I 4 , I 5 , I 6 , I 7 , I 8 Represents 8 images, each of which I i There is a corresponding classification result C(I i );
[0049] Classification confidence greater than or equal to 0.45 is considered to be correct anomalies:
[0050]
[0051] If Confidence(C(I i ))≥0.45, then C(I i) is the correct classification; Confidence(C(I i )) indicates the classification C(I i )’s confidence level;
[0052] If the number of abnormal images of the same type exceeds 5 within 1 second, it is confirmed that the examinee has abnormal behavior: N(C) represents the number of abnormal images of category C, then the logical expression is:
[0053]
[0054] Compared with the prior art, the present invention has the following beneficial effects:
[0055] This paper creatively combines advanced computer vision technology and deep learning algorithms to build a nine-point key part recognition system based on the YOLOv8-pose model, and innovatively designs the QRABPNet multi-classification model that works with it, aiming to achieve real-time monitoring and intelligent analysis of examinees' behavior during the exam. This system effectively overcomes the problems of tight human resources and visual blind spots in traditional invigilation methods, and significantly reduces the missed detection rate of cheating in exams.
[0056] Through in-depth analysis of the examinee's posture, the present invention carefully defines nine key points including nose, right eye, left eye, right shoulder, left shoulder, right elbow, left elbow, right wrist, and left wrist. The state and relative position changes of these key points can accurately reflect the examinee's behavior pattern. The QRABPNet multi-classification model further conducts in-depth analysis of the 18 data dimensions of these key points to identify and detect five abnormal actions including the examinee lying on the table, standing up to peek, leaning to peek, placing both hands under the table, and raising hands.
[0057] The core of the present invention lies in its innovative classification mechanism and alarm system. Through real-time analysis of examinees' behavior, the system can automatically judge abnormal behavior according to preset rules, and immediately trigger the alarm mechanism to alert the invigilator. This process not only takes into account the efficiency and accuracy of recognition, but also greatly improves the efficiency of invigilation work, realizes continuous monitoring of the examination room, effectively reduces the occurrence of cheating, and ensures the fairness and seriousness of the examination.
[0058] In general, the present invention provides an efficient and comprehensive invigilation solution, which not only optimizes the use of invigilation resources and reduces the burden on invigilators, but also ensures the fairness of the examination environment and the concentration of examinees through technological innovation, providing strong support for maintaining the fairness of educational examinations. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the technical solutions of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, what is described below is only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0060] Figure 1 The figure is an overall flow chart of the method for identifying abnormal behavior of examinees of the present invention.
[0061] Figure 2 It is a normalized schematic diagram in an embodiment of the present invention.
[0062] Figure 3 Schematic diagram of a training data set for a custom posture detection model in an embodiment of the present invention.
[0063] Figure 4 This is a schematic diagram of the locations of the nine key points of the present invention.
[0064] Figure 5 Schematic diagram of the training process of a custom posture detection model in an embodiment of the present invention.
[0065] Figure 6 Schematic diagram of training results of a custom posture detection model in an embodiment of the present invention.
[0066] Figure 7 Schematic diagram of model posture evaluation in an embodiment of the present invention.
[0067] Figure 8 This is a schematic diagram of the network structure of the custom QRABPNet network model. DETAILED DESCRIPTION
[0068] The invention will be further described below in conjunction with specific embodiments.
[0069] In order to detect abnormal behaviors of examinees in the examination room, such as lying on the table, standing up and peeking forward, leaning to the left and right, putting both hands under the table, and raising hands, the present invention proposes a method for identifying examinees' abnormal behaviors based on YOLOv8 human posture evaluation. The overall process is as follows: Figure 1 The method includes obtaining the examinee image from the real-time video stream of the monitoring system based on the defined frame extraction rules; performing image stitching, image contrast adjustment, target extraction, and normalization on the extracted test images to complete data preprocessing; inputting the preprocessed examinee image into the custom posture detection model to propose 9 key points of the examinee's posture; inputting the obtained 9 key points into QRABPNet to identify the abnormal row name and confidence; confirming based on the abnormal behavior judgment rules; and pushing abnormal behavior warnings.
[0070] This embodiment takes the abnormal behavior detection scenario of examinees in an examination room as an example to further illustrate the detection method of the present invention.
[0071] 1. Based on the defined frame extraction rules, obtain the candidate's image from the real-time video stream of the monitoring system.
[0072] (1) Time synchronization is performed on network cameras in the examination room to ensure that the time of network cameras in the same examination room is consistent.
[0073] Assume that we have n network cameras in the examination room, and the time synchronization error of each camera is Δt i represents (i from 1 to n), then ideally, the time of all cameras should be synchronized to the same time point t 0 . Therefore, we can get the following formula:
[0074] Δt 1 =Δt 2 =…=Δt n =t 0
[0075] (2) Polling is performed 8 times per second to extract images from the network cameras in the examination room. The images extracted each time are grouped into image groups in the order of the network cameras. By testing each abnormal behavior, the extraction is performed 8 times per second to ensure that there is no missed detection and no redundant detection, thereby improving efficiency.
[0076] The camera polls 8 times in 1 second, and the time interval between each image extraction is equal, set to Δt, then the time point of each image extraction can be expressed as t 0 +kΔt, where k is an integer from 0 to 7. In this way, each polling can get an image group, which contains 8 images, and its time point can be expressed as:
[0077] t 0 ,t 0 +Δt,t 0 +2Δt,…,t 0 +7Δt
[0078] (3) The eight images extracted within one second form an examination room image group, which is then sequenced in time series. The time series images ensure the continuity of the images and are used as the judgment rule for abnormal behaviors that occur eight times in a row in step 5.
[0079] Assuming that the time required for each polling to extract images and perform abnormal behavior detection is T (including image transmission, processing, etc.), then Complete abnormal behavior detection. If 8 detections are required per second, then: Thus we get Seconds, that is, every 8 detection cycles constitute 1 second.
[0080] If we extract 8 images per second to form an image group, then the time point of the k-th image extraction is t 0 +kΔt, and the time point of the k+1th extraction is t 0 +(k+1)Δt. The continuity of the image can be expressed by the difference between two extraction time points, that is: Δt k+1 -Δt k =Δt This ensures the continuity of the image in time series.
[0081] 2. Perform image stitching, image contrast adjustment, target extraction and normalization on the extracted candidate images to complete data preprocessing.
[0082] (1) Image stitching: merge the images from different network cameras in the examination room into one image at the same time. After the cameras are installed in an examination room, their positions, field of view angles, and resolutions remain unchanged without human intervention; therefore, image stitching is used to stitch the images from multiple cameras at the same time into one image. The main functions of image stitching are SIFT_create, BFMatcher, warpPerspective, and copyMakeBorder provided by the OpenCV computer vision library.
[0083] Camera calibration, calibrate each camera to obtain its internal parameters (focal length, principal point, distortion coefficient, etc.) and external parameters (position and orientation of the camera in the world coordinate system).
[0084] Synchronous image acquisition ensures that all cameras capture images at the same time point to avoid dynamic scene inconsistency problems caused by time difference.
[0085] Feature extraction and matching,The SIFT feature extraction algorithm is used to extract feature points from the image of each camera, and the feature matching algorithm finds the corresponding feature points between images of different cameras.
[0086] Image transformation and registration, based on the matched feature points, calculate the homography matrix (HomographyMatrix) or fundamental matrix (Fundamental Matrix) between images. These matrices describe the geometric transformation relationship between images. Use these matrices to transform the images so that images taken by different cameras can be aligned in the same coordinate system.
[0087] Image fusion, after registration, multiple images are fused using Laplacian pyramid fusion.
[0088] Seam line processing: smooth the seam lines of the overlapping areas after image fusion to make the transition more natural.
[0089] (2) Image contrast adjustment: Use contrast-limited adaptive histogram equalization to enhance the image contrast. Reduce noise amplification by limiting the intensity of histogram equalization for each block. It sets a threshold (contrast limit). When the bin in the histogram exceeds this threshold, the excess pixel values will be evenly distributed to other bins. This can avoid over-enhancement and noise amplification. The execution steps are as follows.
[0090] Block segmentation divides the image into several small, non-overlapping areas (blocks).
[0091] Compute the histogram for each block. Compute the histogram for each block separately.
[0092] Contrast limitation, the contrast of the histogram of each block is limited, that is, a maximum threshold is set. When the number of pixels at a certain gray level exceeds this threshold, these excess pixels are evenly distributed to other gray levels.
[0093] Histogram equalization, uses the modified histogram to perform histogram equalization on each block.
[0094] Histogram accumulation, in order to avoid obvious boundaries between blocks, a gradient transition or bilinear interpolation can be used to smooth the histogram between blocks.
[0095] Image reconstruction,reconstructs each block using the equalized histogram, and then merges all blocks into one image.
[0096] (3) Target extraction: Combine the test room calibration data to distinguish the corridor area, vacant area, and test camera area, and generate an image for each test camera.
[0097] Calibration data matching: Match the calibration data with the spliced image and calculate the width and height matching coefficients (W n , H n ). Take the upper left corner of the image as the reference point (TX 0 ,TY 0 ), width (TW), height (TX), and the upper left corner of the calibration data (BX 0 , BY 0 ), width (BW), and height (BH) are calculated using the following formula.
[0098]
[0099] Image extraction: Use the matching coefficient to calculate the actual coordinates of each corridor area, vacant area, and test area, and use Opencv's getRectSubPix function to extract the sub-image from the rectangular area. Determine the upper left corner of the sub-area (BX 0 ,BY 0)、lower right corner (BX 1 ,BY 1 ), corresponding to the upper left corner of the image coordinates (TX 0 ,TY 0 ), lower right corner (TX 1 ,TY 1 ), and the calculation formula is as follows.
[0100] TX n =W n ×BX n Among them TX n Represents the X coordinate of the calibration point to be determined, BW n Represents the original X coordinate of the calibration point.
[0101] TH n =H n ×BH n where TH n represents the X coordinate of the calibration point to be determined, BH n Represents the original X coordinate of the calibration point.
[0102] (4) Normalization: to achieve uniform processing of the size of the acquired test camera images. The present invention processes the test camera images uniformly into 640×480 images. The effect is as follows: Figure 2 shown.
[0103] Calculate scaling: Calculate the aspect ratio of each test camera image, scale the image according to the ratio, and unify it into 640×480 images. In the scaling process, scale_width is the width scaling ratio, scale_height is the height scaling ratio, width is the original width of the image, and height is the original height of the image.
[0104] The image is wider than it is tall:
[0105]
[0106] Image width is less than or equal to height:
[0107]
[0108] Scale the image:
[0109] Use Opencv's resize function to scale and merge into a 640×480 image. This process saves the image normalization step when using yolov8 for inference, which improves the speed.
[0110] 3. Input the preprocessed candidate image into the custom posture detection model to extract 9 key points of the candidate's posture.
[0111] The custom posture detection model is based on yolov8n-pose.pt to train the posture model of the examinee in the examination room, which is used to detect 9 key points, including nose, left eye, right eye, left shoulder, right shoulder, left elbow, right elbow, left wrist, and right wrist. Figure 3 As shown, the marking method is as follows Figure 4 The model training process is as follows.
[0112] (1) Dataset annotation. We collected 300 images of each type of abnormal behavior, such as lying on the table, standing up and looking forward, leaning to the left and right, placing both hands under the table, and raising hands, for a total of 1,500 images. We used Labelme for annotation; first, we annotated the examinee’s figure, and then annotated the nine key points.
[0113] (2) Model training
[0114] Table 1 Training parameter settings
[0115] Serial number Parameter name Setting Value Remark 1 epochs 100 Number of training rounds 2 patience 10 Number of waiting rounds for early stopping 3 batch -1 The number of images in each batch 4 imgsz 640 Input image size 5 save_period 10 The interval for saving checkpoints 6 cache True Whether to use cache when loading data 7 cache True Whether to use cache when loading data 8 cos_lr True Use a cosine learning rate scheduler 9 lr0 0.01 Initial learning rate 10 lrf 0.01 Final learning rate 11 pose 13.0 Pose loss weight 12 kobj 2.0 Keypoint target loss weight 13 val True Whether to perform validation / testing during training 14 optimizer Lion Select the optimizer to use
[0116] Training parameter settings, refer to Table 1, optimizer customization improvements:
[0117] Lion was introduced when selecting the optimizer, which reduced memory usage and improved efficiency. First, install lion-pytorch; second, import the Lion optimizer in trainer.py and set the parameters optimizer = Lion (g[2], lr = lr, betas = (momentum, 0.9999), weight_decay = decay); finally, select the Lion optimizer during training, optimizer = "Lion".
[0118] The training process is as follows Figure 5 As shown in Figure 2, the model training results are as follows: Figure 6 shown.
[0119] (3) Model evaluation. We randomly selected 100 images to test the best.pt obtained after training. We drew the human body outline, 9 key points, and drew connecting lines for the key points of the head and upper body. Some of the results are shown below. Figure 7 shown.
[0120] 4. Input the obtained 9 key points into QRABPNet to identify the abnormal row name and confidence.
[0121] The QRABPNet network model receives the 9 key points of each candidate image output in step 3, outputs the abnormal behavior classification and confidence level shown in the test image, and the results are used for judgment in step 5. The QRABPNet network model training process is as follows.
[0122] (1) Key point data acquisition. 200 pictures of abnormal behaviors, including lying on the table, standing up and looking forward, leaning sideways and looking left and right, placing both hands under the table, and raising hands, were collected and stored in different folders according to the abnormality category.
[0123] The model trained in step 3 is used to obtain the key points of nose, left_eye, right_eye, left_shoulder, right_shoulder, left_elbow, right_elbow, left_wrist, and right_wrist of the human body in each picture, and the image name and category are written into CSV to form a data set. The column definition of the CSV file includes image_name, label, nose_x, nose_y, left_eye_x, left_eye_y, right_eye_x, right_eye_y, left_shoulder_x, left_shoulder_y, right_shoulder_x, right_shoulder_y, left_elbow_x, left_elbow_y, right_elbow_x, right_elbow_y, left_wrist_x, left_wrist_y, right_wrist_x, and right_wrist_y.
[0124] (2) Network model definition. Define the QRABPNet neural network class to implement multi-classification of abnormal behavior key point data. The QRABPNet class inherits from the basic class nn.Module for building neural networks. The specific structure is as follows: Figure 8As shown. According to the coordinate data of 9 key points, the number of input features is calculated to be 18, and the output categories are 5 (lying down on the table, standing up and peeking forward, leaning to the left and right and peeking, placing both hands under the table, and raising hands). First, create a fully connected layer to receive the initialized number of features, convert the input data into a linear combination of 256 features; and use the activation function ReLU to introduce nonlinear factors to help the network learn more complex features. Secondly, create the second fully connected layer of the network, the number of input features is 256, the number of output features is 512, and create a ReLU activation function for use after the second fully connected layer. Again, create the third fully connected layer of the network, the number of input features is 512, the number of output features is 256, and create a ReLU activation function for use after the third fully connected layer. Finally, by creating the fourth fully connected layer, the number of input features is 256, and the number of output features is num_classes, that is, the number of categories defined. Define forward for training and inference to implement the forward propagation function. QRABPNet has a neural network with four hidden layers, and each hidden layer uses the ReLU activation function for nonlinear transformation.
[0125] (3) Model training. Set batch_size to 18, learning_rate to 0.01, and num_epochs to 100, and the optimal result is a Loss of 0.0667 and an Acc of 97%.
[0126] 5. Confirm based on abnormal behavior determination rules.
[0127] Based on the classification information obtained in step 4, the analysis results of 8 images within 1 second are accumulated; the classification confidence greater than or equal to 0.45 is identified as a correct anomaly; if the same type of anomaly accumulates for more than 5 images within 1 second, it is confirmed that the candidate has abnormal behavior.
[0128] The analysis results of 8 images in 1 second: I 1 , I 2 , I 3 , I 4 , I 5 , I 6 , I 7 , I 8 Represents 8 images, each of which I i There is a corresponding classification result C(I i ).
[0129] Classification confidence greater than or equal to 0.45 is considered to be correct anomalies: If Confidence(C(I i ))≥0.45, then C(Ii ) is the correct classification. i )) indicates the classification C(I i ) confidence level.
[0130] If the number of abnormal images of the same type exceeds 5 within 1 second, it is confirmed that the examinee has abnormal behavior: N(C) represents the number of abnormal images of category C, then the logical expression is:
[0131] 6. Push notifications for abnormal behavior warnings.
[0132] Actively push information judged as abnormal behavior to the configured suspected alarm receiving address.
[0133] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0134] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A method for identifying abnormal behavior of examinees based on human posture assessment, characterized in that: The process includes: Step 1: Based on the defined frame extraction rules, the examinee image is obtained from the real-time video stream of the monitoring system; Step 2, preprocessing the extracted candidate images to obtain standard candidate images; Step 3, obtain the standard candidate image, and obtain the key points of the candidate's posture through the custom posture detection model, including nose, left eye left_eye, right eye right_eye, left shoulder left_shoulder, right shoulder right_shoulder, left elbow left_elbow, right elbow right_elbow, left wrist left_wrist, right wrist right_wrist, a total of 9 key points; Step 4, obtain 9 key point data, and identify abnormal behavior names and confidence levels through the custom classification model QRABPNet model; the abnormal behavior names include 5 categories: lying on the table, standing up and peeking forward, leaning sideways and peeking left and right, placing both hands under the table, and raising hands; The custom classification model QRABPNet model is a multi-classification of abnormal behavior key point data; the QRABPNet model is based on the basic class nn.Module for building neural networks, and calculates the number of input features as 18 and the number of output categories as 5 according to the coordinate data of 9 key points; first, a fully connected layer is created to receive the initialized number of features, and the input data is converted into a linear combination of 256 features; and the activation function ReLU is used to introduce nonlinear factors to help the network learn more complex features; secondly, a second fully connected layer of the network is created, with the number of input features being 256 and the number of output features being 512, and Create a ReLU activation function for use after the second fully connected layer; again, create the third fully connected layer of the network, the number of input features is 512, the number of output features is 256, and create a ReLU activation function for use after the third fully connected layer; finally, by creating a fourth fully connected layer, the number of input features is 256, and the number of output features is num_classes, that is, the number of defined categories; define forward for training and reasoning to implement forward propagation functions; the QRABPNet model has a neural network with four hidden layers, and each hidden layer uses the ReLU activation function for nonlinear transformation; Step 5: confirm whether abnormal behavior exists based on the abnormal behavior determination rules. If so, push an abnormal behavior warning.
2. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 1, characterized in that: The n cameras in the examination room cover the entire examination room. Each camera has a synchronization time and time interval. The network cameras in the examination room are polled 8 times every second and images are extracted at the same time. The images extracted each time are organized into an image group according to the order of the network cameras, which are used for the key point extraction of the examinee's posture in step 3; the 8 images extracted within 1 second form an examination room image group, which are formed into a sequence group according to the time sequence. The time sequence images ensure the continuity of the images and are used for abnormal behavior judgment in step 5.
3. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 1, characterized in that: The preprocessing process in step 2 includes image stitching, candidate target extraction and normalization processing in sequence; the image stitching is to merge the images of different network cameras in the examination room at the same time into one image; the candidate target extraction is to generate an image according to each examination camera position; the normalization processing is to unify the size of the acquired examination camera position images.
4. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 3, characterized in that: The image stitching and candidate target extraction also include the process of image contrast adjustment, which is specifically: S11, block division, divides the image into several small, non-overlapping area blocks; S12, calculating the histogram of each block, and calculating the histogram for each block separately; S13, contrast limitation, performs contrast limitation on the histogram of each block, that is, sets a maximum threshold. When the number of pixels at a certain gray level exceeds this threshold, these redundant pixels are evenly distributed to other gray levels. S14, histogram equalization, performing histogram equalization on each block using the modified histogram; S15, histogram accumulation, in order to avoid obvious boundaries between blocks, use gradient transition or bilinear interpolation to smooth the histogram between blocks; S16, image reconstruction, reconstructs each block using the equalized histogram, and then merges all blocks into one image.
5. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 3, characterized in that: The image stitching is specifically as follows: S21, camera calibration, calibrating each camera to obtain its internal and external parameters; S22, synchronously collecting images to ensure that all cameras collect images at the same time point; S23, feature extraction and matching, using SIFT feature extraction algorithm to extract feature points from the image of each camera, and feature matching algorithm to find corresponding feature points between images of different cameras; S24, image transformation and registration, calculating the homography matrix or basic matrix between images according to the matched feature points, these matrices describe the geometric transformation relationship between images, and using these matrices to transform the images so that the images taken by different cameras can be aligned in the same coordinate system; S25, image fusion, fusing multiple images after registration using Laplacian pyramid fusion; S26, seam line processing, smoothing the seam line of the overlapping area after the image fusion, so that the transition is more natural.
6. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 3, characterized in that: The candidate target extraction is to distinguish the corridor area, the vacant area, and the test camera area in combination with the test room calibration data, and generate an image according to each test camera; specifically, it includes: Calibration data matching: Match the calibration data with the spliced image and calculate the width and height matching coefficients (W n , H n ); Take the upper left corner of the image as the reference point (TX0, TY0), width TW, height TX and the upper left corner of the calibration data (BX0, BY0), width BW, height BH for calculation, the formula is as follows: Image extraction: Use the matching coefficient to calculate the actual coordinates of each corridor area, vacant area, and test area, and use Opencv's getRectSubPix function to extract sub-images from the rectangular area; calibrate the upper left corner (BX0, BY0) and lower right corner (BX1, BY1) of the sub-area, and the corresponding image coordinates are the upper left corner (TX0, TY0) and lower right corner (TX1, TY1). The calculation formula is as follows: TX n Represents the X coordinate of the calibration point to be found, BW n Represents the original X coordinate of the calibration point; where TH n represents the X coordinate of the calibration point to be determined, BH n Represents the original X coordinate of the calibration point.
7. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 3, characterized in that: The normalization process is to unify the size of the acquired test camera images; including: Calculate scaling: Calculate the aspect ratio of each test camera image, scale the image according to the ratio, and unify it into 640×480 images; during the scaling process, scale_width is the width scaling ratio, scale_height is the height scaling ratio, width is the original width of the image, and height is the original height of the image; The image is wider than it is tall: Image width is less than or equal to height: Scale the image: Use Opencv's resize function to scale and merge into a 640×480 image.
8. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 1, characterized in that: The custom posture detection model in step 3 is a candidate posture model trained based on Yolov8n-pose.pt.
9. The method for identifying abnormal behavior of examinees based on human posture assessment as claimed in claim 1, characterized in that: The abnormal behavior determination rules in step 5 are specifically: Based on the classification information obtained in step 4, the analysis results of 8 images within 1 second are accumulated; the classification confidence greater than or equal to 0.45 is identified as a correct anomaly; If more than 5 abnormal pictures of the same type are counted within 1 second, it is confirmed that the examinee has abnormal behavior; The analysis results of 8 images within 1 second are as follows: , , , , , , , Represents 8 images, each of which There is a corresponding classification result ; Classification confidence greater than or equal to 0.45 is considered to be correct anomalies: Correct classification if but is the correct classification; Representation classification confidence level; If the same type of abnormality accumulates more than 5 pictures within 1 second, it is confirmed that the candidate has abnormal behavior: Indicates category The number of abnormal images, then the logical expression is: Abnormal behavior .