An end-to-end examination room behavior analysis method and system

Abstract

This invention discloses an end-to-end examination behavior analysis method and system, comprising a video input management module, a video decoding module, an image preprocessing module, a raw frame prediction queue, a raw frame queue, an AI module, an analysis result recording module, and an abnormal event video generation module. It utilizes an end-to-end AI model training and inference scheme, enabling efficient training using large amounts of historical data. In practice, it offers fast processing speed and high efficiency. The end-to-end AI model takes continuous video frames as input and outputs bounding boxes indicating suspected violations after inference. In the post-processing section, the model output undergoes spatiotemporal correlation and filtering to finally generate spatiotemporal location information of suspected violations. It exhibits good scene adaptability; when new scenes are needed or new actions to be identified are added, only new training data needs to be added and retrained to expand the algorithm, saving manpower and improving efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of examinee behavior detection, and particularly relates to an end-to-end examination room behavior analysis method and system. BACKGROUND

[0002] In order to maintain the seriousness of the examination, video monitoring invigilation has become a routine requirement in the examination room. In the examination, a large amount of data is retained by the video, and the efficiency of manual screening of the video content is low, and it is difficult to control the standardized screening accuracy. This year, the video analysis technology under the development of neural networks has been greatly improved, and computer-aided video violation analysis has become possible. Some methods use features based on skeletal feature points for behavior analysis. However, this method has some problems in use:

[0003] 1. The examinee's irregular behavior categories are various, and it is difficult to extend the action of multiple categories by using the method of skeletal points.

[0004] 2. Affected by the video angle, the skeletal point distribution of some seats under the oblique view angle is different, and it is impossible to use a unified formula to represent the classifier of cheating actions.

[0005] 3. The method based on skeletal points needs a complex post-frame processing process, and a large amount of parameter optimization and adjustment work.

[0006] The present method bypasses the above-mentioned skeletal point extraction, and uses an end-to-end model method scheme, inputs a short video sequence, extracts the spatial and temporal information of the video information through two branch convolutions, and locates the pre-defined actions. SUMMARY

[0007] To solve the above technical problems, the present application provides an end-to-end examination room behavior analysis method, comprising the following steps:

[0008] Obtaining real-time video data collected, and obtaining a single frame image according to the real-time video data;

[0009] Identifying and sample labeling a target object in the single frame image through a sample labeling tool;

[0010] Labeling the single frame image according to the target object, and obtaining at least one target image;

[0011] Using a target tracking method to extract feature data of the target object in the target image, and position coordinates of an event occurrence on each frame image between the start and end frames of the event;

[0012] Screening the target image of the target object as the examinee, obtaining a continuous video segment according to a single frame image corresponding to the target image, performing interactive aggregation according to the single frame image and the continuous video segment, and exporting all labeled data as negative samples and corresponding to the original video;

[0013] Artificially labeling events and exporting the labeling results.

[0014] Preferably, the sample labeling is to artificially label the above-mentioned five kinds of rule violations in the video, including the position of each frame image from the start to the end of the rule violation and the category of the rule violation.

[0015] Preferably, the single frame image is extracted at 3 frames per second, and the extracted picture is saved as a sample to be labeled.

[0016] Preferably, the sample labeling tool labels events for the imported image sequence, and a complete event includes a start frame Frame.start, an end frame Frame.stop, and the position coordinates of the event occurrence on each frame image between the start and end frames.

[0017] Preferably, the negative sample refers to the normal action of the examinee during the examination, which is quantitatively described as follows:

[0018] The time sequence of the person (examinee) in the image without being labeled as a rule violation;

[0019] The action sequence with an overlap interval ROI of less than 50% with the labeled action;

[0020] The area in the image other than the person.

[0021] Preferably, the loss function is designed as follows:

[0022] The loss function mainly includes two parts: one part is the loss LCls of the event category, which is used to measure the correct classification of the event;

[0023] The other part is the loss LD of the actual position, which is used to measure the correct labeling of the position of the event.

[0024] An end-to-end examination room behavior analysis system, which is composed of a video input management module, a video decoding module, an image preprocessing module, an original frame to be predicted queue, an original frame queue, an AI module, an analysis result recording module, and an abnormal event video generation module.

[0025] Preferably, the video input management module is used to receive and manage the real-time video stream to be analyzed set by the user. The module connects the video source in the user set protocol mode, disassembles the control data in the data stream, and sends the video data stream therein to the video decoding module through the shared memory block.

[0026] Preferably, the video decoding module accepts the real-time video stream from the shared memory of the video input management module, detects the original encoding format and resolution of the video stream, and frame rate video information.

[0027] Preferably, the AI module includes AI inference initialization, data input control, AI prediction inference, inference result decoding, prediction result merging and filtering, result output function, and specific operation steps:

[0028] S1, initialize an empty event list.

[0029] S2, when a new detection result is obtained, perform IOU operation on each existing result in the queue, if the spatial overlap ratio is greater than 0.5, merge the two events (x1, y1, x2, y2, conf, type) (x1', y1', x2', y2', conf', type') into an event with coordinates (min(x1, x1'), min(y1, y1'), max(x2, x2'), max(y2, y2')) and confidence max(conf, conf') class label as the event with greater confidence.

[0030] If the spatial overlap ratio is less than 0.5, add the new detection to the list;

[0031] S3, repeat step 2 until all events are put into the list;

[0032] S4, calculate the coordinates in the list events as original coordinate information according to the transformation ratio, and sequentially push into the message queue connected with the analysis result recording module. Each record contains the following information (frame number, xo, yo, xo', yo', confidence, type);

[0033] S5, the analysis result recording module obtains all marked events from the message queue and records them in the database. According to the frame number, the original frame and several frames after it are obtained from the image queue, and the event position is marked on the original picture according to the coordinates (xo, yo, xo', yo'), and the marked frame is sequentially pushed into the video compression module to generate an abnormal event video with annotation.

[0034] The technical effects and advantages of the present application are:

[0035] 1、The end-to-end AI model training and inference scheme used in the application can efficiently train through a large amount of historical data, and has high processing speed and efficiency in use. The end-to-end AI model takes continuous video frames as input and outputs a rectangular box prompt of suspected illegal behavior after inference. In the post-processing part, the model output is correlated and filtered in the time and space domains, and finally the time and space position information of the suspected illegal behavior is generated.

[0036] 2、The end-to-end model training and inference scheme is simple and clear, and the model training efficiency is high.

[0037] 3、The scene adaptability is good. When a new scene is needed or a new action to be recognized is added, only new training data needs to be added for retraining, which saves manpower and improves efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the network structure diagram of the end-to-end examination room behavior analysis system provided by the embodiment of the application. DETAILED DESCRIPTION

[0039] The application will be further described in detail below in combination with the drawings and specific embodiments. The embodiments of the application are given for the purpose of illustration and description, and are not exhaustive or limit the application to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical application of the application, and to enable those of ordinary skill in the art to understand the application so as to design various embodiments with various modifications suitable for specific purposes.

[0040] Embodiment 1

[0041] Please refer to Figure 1 In this embodiment, an end-to-end examination room behavior analysis method is provided. The input is divided into two branches. In the 3D CNN network branch, continuous images are used as model input, mainly for extracting spatial and temporal information of behavior. The dimension of the input image is [3xDxHxW], wherein: 3 represents that the image is a 3-channel color (RGB) image, D represents the number of images, usually 8 according to the occurrence of an action, H represents the height of the image, and W represents the width of the image. The input image is usually 1280x720.

[0042] 3D CNN network part, the case uses ResNext50 3D convolutional neural network to extract spatial and temporal domain feature information, and the dimension of the output feature vector is [C'D'H'W'], wherein: C' is the number of output channels, D' = 1, H' = H / 32, and W' = W / 32.

[0043] Another branch 2D CNN, with the middle frame image as input, this input branch is mainly used to extract the spatial position information of the behavior information. The dimension of the input image is [3xHxW], and the feature information in the space-time domain is extracted through the CSPDarknet53 network structure. The dimension of the output feature vector is [C”xD”xH”xW”], where: C” is the number of output channels, D”=1, H”=H / 32, W”=W / 32

[0044] The feature layer output by the 3D CNN branch is connected with the feature layer output by the 2D CNN branch as the result of feature extraction. After mixing the output results of the two branches, 2D CNN feature extraction is continued, and in this case, the Neck and Head of YOLOV4 are used.

[0045] The final output is also divided into two branches, one branch classifies the recognized behaviors, and the other branch is used to recursively calculate the spatial position of the behavior.

[0046] Among them, the original video is partly from the video recorded in the actual examination, and partly from the artificially made video containing various cheating behaviors. A total of 100 videos are used in this case, of which 88 are used as training videos and 12 are used as test videos. The total video length is 56,249 seconds, about 1,406,225 images. It contains five kinds of illegal actions: raising hands, standing, turning head, stretching hands and other suspicious behavior events (positive samples) marked:

[0047] Sample labeling is to manually mark the above-mentioned 5 kinds of illegal actions in the video, including the position (BBox) of each frame image from the beginning to the end of the illegal action and the category of the illegal action. The manual labeling process includes:

[0048] Sample the original video to get a fixed time interval image sequence;

[0049] Import the image into the labeling tool;

[0050] Manually label the event (Action, referring to the above-mentioned various illegal actions);

[0051] Export the labeling results.

[0052] In order to reduce the workload of manual labeling, the original video is down-sampled in this case. Generally, the video frame rate is 25 frames per second, and we sample 3 frames per second. Save the sampled pictures as samples to be labeled.

[0053] Use the sample labeling tool to label the imported image sequence. A complete event includes the start frame Frame.start, the end frame Frame.stop, and the position coordinates (BBoxes) of the event on each frame image between the start and end frames.

[0054] Export all the labeled data, and correspond to the original video.

[0055] Negative sample generation:

[0056] Negative samples refer to the normal actions of the examinee during the examination. The quantitative description is as follows:

[0057] The time sequence of the person (examinee) in the image without being labeled as a violation action

[0058] Action sequence with ROI <50% overlap with the labeled action.

[0059] Other areas in the image except for the person.

[0060] Loss function (Loss) design:

[0061] The loss function mainly includes two parts: one part is the event class loss LCls, which is used to measure the correct classification of events, and the other part is the actual position loss LD, which is used to measure the correct marking of the position of the event. The total loss is obtained by adding the two parts, as follows:

[0062] L final =λL D +L Cls

[0063] Where λ = 0.5, LD is the detection error, and LCls is the classification error.

[0064] L D =L x +L y +L w +L h +L con x

[0065] In the above formula, L1 Smooth Loss is used to calculate the loss of x, y, w, and h respectively.

[0066] The formula for calculating L1 Smooth Loss is:

[0067]

[0068] Where x is the predicted result, and y is the ground truth value.

[0069] For Lconf in it, Lmse Loss is used, which is defined as:

[0070] L MSE (x,y)=(x-y) 2

[0071] LCls uses Focal Loss, defined as:

[0072] L foal (x, y) = y(1 - x) γ log(x) + (1 - y)x γ log(1 - x)

[0073] Training process:

[0074] First, we independently initialized two network branches of 2D and 3D. Among them, the 3D network, we used the pre-training weight on Kinetics; the 2D network branch used the pre-training weight on PASCAL VOC dataset. We used the Stochastic Gradient Decent (SGD) method and weight decay strategy to optimize the loss function. The initial Learning Rate (LR) was selected as 1e-05, and after every 10 Epoch training, the learning rate was reduced to 0.5 of the original. The complete architecture was trained on 4 NVIDIA 1080TI GPUs using the Pytorch end-to-end training framework, and after 40 Epoch, the loss function reached a stable value.

[0075] Example 2

[0076] In this embodiment, an end-to-end examination room behavior analysis system is provided, which is composed of a video input management module, a video decoding module, an image preprocessing module, an original frame to be predicted queue, an original frame queue, an AI module, an analysis result recording module, and an abnormal event video generation module.

[0077] The video input management module is used for receiving and managing the real-time video stream to be analyzed set by the user. The module connects the video source in the user-set protocol mode, disassembles the control data in the data stream, and sends the video data stream therein to the video decoding module through the shared memory block. The user can select the connection protocol, including: video file, SIP, RTSP, GB28181, and TCP mode.

[0078] The video decoding module accepts the real-time video stream from the shared memory of the video input management module, detects the original encoding format and resolution of the video stream, and frame rate video information. The video stream is decoded by a corresponding decoder to restore the original image data. The decoder module includes H264, H265, webm, and resolutions including 360P, 720P, 1080P, and 2K video. After decoding by the original video decoding module, one path is pushed into the original frame pair; another path performs time domain downsampling and spatial domain image size normalization on the image, extracts video key frames at a rate of 3 frames per second according to the detected video frame rate, resizes the key frames to the input size of the inference module, i.e., 1280x720, and records the transformation ratio ratio. The key frames and corresponding frame numbers are pushed into the tail of the image to be predicted memory queue.

[0079] The AI module includes AI inference initialization, data input control, AI prediction inference, prediction result decoding, prediction result merging and filtering, and result output functions. The AI weight file is the output result of the training described in the method section. In the AI inference machine initialization stage, the AI inference machine initializes the model structure and loads the AI weight file. In this case, the nvidia triton inference service engine is used as the inference machine. The data input control part monitors the queue length of the frame to be predicted queue. When the data length is greater than the selected frame length of 8 frames, 8 frames of data are taken from the head of the frame to be predicted queue, and the data is spliced into an array form as the input of the AI inference machine. Four frames of image are popped from the top of the queue; thus, the next time data is obtained, 4 frames of data are the data predicted last time, and 4 frames of data are new data, i.e., 50% of the data overlaps between two inferences, which can better avoid the problem of missing detection due to the time of action and the sampling time not being matched. The AI prediction inference machine performs inference prediction on the input data according to the loaded model and weight, and generates the prediction results of the two branches as described in the method. The prediction results are decoded as described in the method to obtain behavior classification and confidence (type, confidence), BBox position information (x1, y1, x2, y2), where x1, y1 are the coordinates of the upper left corner of the bbox, and x2, y2 are the coordinates of the lower right corner of the bbox.

[0080] The prediction result merging and filtering includes confidence filtering and overlapping event merging functions. The confidence filtering function discards the results below the confidence threshold according to the preset system sensitivity confidence threshold. The default confidence used in this case is 0.5. The overlapping event merging function merges the events with overlapping positions in the same frame into one event.

[0081] Specific operation steps:

[0082] S1, initialize an empty event list.

[0083] S2, when there is a new detection result, IOU operation is performed between the new detection result and each existing result in the queue, if the spatial overlap ratio is greater than 0.5, the two events of (x1, y1, x2, y2, conf, type) and (x1', y1', x2', y2', conf', type') are merged into an event with coordinates (min(x1, x1'), min(y1, y1'), max(x2, x2'), max(y2, y2')) and the event with greater confidence is marked as the class.

[0084] If the spatial overlap ratio is less than 0.5, the new detection is added to the list;

[0085] S3, repeat step 2 until all events are put into the list;

[0086] S4, the coordinates in the list events are calculated as original coordinate information according to the transformation ratio, and are sequentially pushed into the message queue connected with the analysis result recording module. Each record contains the following information (frame number, xo, yo, xo', yo', confidence, type);

[0087] S5, the analysis result recording module obtains all marked events from the message queue and records them in the database. The original frame and several frames after it are obtained from the image queue according to the frame number, and the event position is marked on the original picture according to the coordinates (xo, yo, xo', yo'), and the marked frame is sequentially pushed into the video compression module to generate an abnormal event video with annotations.

[0088] Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art and related fields without creative labor should belong to the scope of protection of the present application. The structures, devices and operation methods not specifically described and explained in the present application, if not specially described and limited, are implemented according to the conventional means in the art.

Claims

1. An end-to-end examination room behavior analysis method, characterized in that, The method comprises the following steps: Obtaining real-time video data collected, and acquiring a single frame image according to the real-time video data; Identifying and sample labeling a target object in the single frame image through a sample labeling tool; Labeling the single frame image according to the target object, and acquiring at least one target image; Using a target tracking method to extract feature data of the target object in the target image and position coordinates of an event occurring on each frame image between a start frame and an end frame of the event; Filtering out a target image in which the target object is a test taker, acquiring a continuous video segment according to a single frame image corresponding to the target image, and performing interactive aggregation on the single frame image and the continuous video segment, and exporting negative samples from all labeled data, the negative samples referring to normal actions of the test taker during the test, and being quantitatively described as follows: The test taker in the image has no time sequence of actions labeled as irregular actions; An action sequence in which an overlap interval ROI with a labeled action is less than 50%; An area in the image other than a human being; Artificially labeling an event, and exporting a labeling result.

2. The end-to-end examination behavior analysis method of claim 1, wherein, The sample labeling is to manually label five kinds of irregular actions in a video, including positions of each frame image from the start to the end of the irregular actions and categories of the irregular actions.

3. The end-to-end examination behavior analysis method of claim 2, wherein, The single frame image is frame-extracted at a rate of 3 frames per second, and the frame-extracted picture is saved as a sample to be labeled.

4. The end-to-end examination behavior analysis method of claim 3, wherein, The sample labeling tool labels events of an imported image sequence, and a complete event includes a start frame Frame.start, an end frame Frame.stop, and position coordinates of the event occurring on each frame image between the start frame and the end frame.

5. The end-to-end examination behavior analysis method of claim 4, wherein, Design of a loss function: The loss function mainly includes two parts: one part is an event category loss LCls used to measure whether an event is correctly classified; and the other part is an actual position loss LD used to measure whether a position of the event is correctly labeled. The video input management module, the video decoding module, the image preprocessing module, the original frame to be predicted queue, the original frame queue, the AI module, the analysis result recording module, and the abnormal event video generation module are combined.

6. An end-to-end examination behavior analysis system using the end-to-end examination behavior analysis method according to any one of claims 1 to 5, characterized by, The video input management module is used for receiving and managing real-time video streams to be analyzed set by a user, the module is connected with a video source in a protocol manner set by the user, control data in a data stream is disassembled, and a video data stream is sent to the video decoding module through a shared memory block.

7. The end-to-end proctoring system of claim 6, wherein, The video decoding module receives real-time video streams from a shared memory of the video input management module, detects original encoding formats, resolutions, and frame rates of the video streams.

8. The end-to-end proctoring system of claim 7, wherein, The AI module includes AI inference initialization, data input control, AI prediction inference, inference result decoding, prediction result merging and filtering, result output functions, and specific operation steps are as follows:

9. The end-to-end proctoring system of claim 8, wherein S1, initializing an empty event list; ​ S2, when there is a new detection result, calculate IOU with each existing result in the queue, if the spatial overlap ratio is greater than 0.5, then merge the two events (x1, y1, x2, y2, conf, type) (x1', y1', x2', y2', conf', type') into one event with coordinates (min(x1, x1'), min(y1, y1'), max(x2, x2'), max(y2, y2')) and the confidence is max(conf, conf') and the class label is the event with greater confidence; If the spatial overlap ratio is less than 0.5, then add the new detection to the list; S3, repeat step 2 until all events are put into the list; S4, calculate the coordinates in the list events into the original coordinate information according to the transformation ratio, and push them into the message queue connected to the analysis result recording module in sequence. Each record contains the following information (frame number, xo, yo, xo', yo', confidence, type); S5, the analysis result recording module obtains all the marked events from the message queue and records them in the database; and obtains the original frame and several frames after it from the image queue according to the frame number, marks the event position on the original picture according to the coordinates (xo, yo, xo', yo'), and pushes the marked frame into the video compression module in sequence to generate an abnormal event video with annotations.

Images